Downtime induced by git access hitting the NFS server

Today at 10:41 UTC we got a huge spike in load in NFS servers. This spike came from https+git and by the look at the graphs it looks like we had a spike in ingress ssh git before that.

On this data I think we need to investigate if this came from CI, and in case it did, we need to work on throttling git access through https to limit how many clients can get in at any time.

Screen_Shot_2017-03-13_at_12.10.41

Screen_Shot_2017-03-13_at_12.09.50

Screen_Shot_2017-03-13_at_12.09.57

Screen_Shot_2017-03-13_at_12.10.06

Screen_Shot_2017-03-13_at_12.10.18

Screen_Shot_2017-03-13_at_12.10.31

Screen_Shot_2017-03-13_at_12.10.36

Screen_Shot_2017-03-13_at_12.10.47

Screen_Shot_2017-03-13_at_12.10.52

Screen_Shot_2017-03-13_at_12.11.02

Screen_Shot_2017-03-13_at_12.11.13

Screen_Shot_2017-03-13_at_12.11.22

Screen_Shot_2017-03-13_at_12.11.30

Screen_Shot_2017-03-13_at_12.13.22

Screen_Shot_2017-03-13_at_12.38.52

Screen_Shot_2017-03-13_at_12.38.59

By the look of the processes monitoring it seems that it came through ssh, but from the HAProxy perspective - this means that the processes metrics are broken-ish.

I'm investigating if this is related to CI builds somehow.