Increase in 5XX errors on GitLab.com 2018-05-01

Context

5XX errors on GitLab.com. Seems to be related to an outage on nfs-file-16.stor.gitlab.com.

Timeline

On date: 2018-05-01

• At 03:15 UTC, nfs-file-16 paged as unresponsive. Upon checking the Azure portal, the server was in a "starting" state. It took some time for the server to start back up, but once it did everything recovered as expected. I've opened a support ticket with Azure to find the root cause of this issue. This issue will be updated once we know more.
• 08h20 UTC - traffic on nfs-file-16.stor.gitlab.com started trending down
• 08h21 UTC - Linux kernel soft lockups start occurring: May 1 08:21:14 nfs-file-16.stor.gitlab.com kernel: [12016.184004] watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
• 08h21 UTC - syslog messages related to TCP timeouts, consul peer failures, etc start appearing
• 08h33 UTC - users start reporting errors
• 08h34 UTC - nfs-file-16.stor.gitlab.com not responding
• 08h52 UTC - @jarv spots that the host is using "problematic" Linux kernel, requests @ahmadsherif rebuild the machine with older kernel.

cc @ahmadsherif