Creating the issue for the record. Timeline: - 20:50 UTC -- PagerDuty alerted that nfs-10 is down - 20:51 UTC -- ack, investigation started - 20:52 UTC -- I was able to ssh in, see the load of ~600 and growing, with about 250 nfsd processes in a locked state. - 20:55 UTC -- alert cleared, number of locked processes started to go down Graphs: - fleet overview: https://performance.gitlab.net/dashboard/db/fleet-overview?orgId=1&from=1514580374689&to=1514581037583 Slack discussion start: https://gitlab.slack.com/archives/C101F3796/p1514580653000183 Errors in dmesg appeared 2 minutes after it was rebooted: ``` [ 240.404065] INFO: task nfsd:1876 blocked for more than 120 seconds. [ 240.406347] Not tainted 4.4.0-104-generic #127-Ubuntu [ 240.408273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ```