2018-06-14: Errors on GitLab resulting in slow page loads or 502s
At approximately 14:10UTC on 2018-06-14 for ~10minutes GitLab was severely degraded.
At this time the root cause appeared to be a large number of users sendings requests to a project on nfs-13. This caused a load spike on the server which then caused multiple nfs timeouts which in turn made the entire site unresponsive.
2018-06-14 15:30 UTC - A change was made to block the problematic requests in our load balancers.
incident doc (internal only): https://docs.google.com/document/d/1B7pyJTv6HKPs5bBWWAjIYUKCb1i_UU3MQwh4D4e71vk/edit#heading=h.26c8go6r6hs
Edited by Dave Smith