Gitaly performance problems on some servers

Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.

Summary

Gitaly performance problems on some servers leading to growing sidekiq queues.

Root Cause Analysis issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7479

image

Service(s) affected : ~"Service:Gitaly" ~"Service:Sidekiq"

Team attribution :

Minutes downtime or degradation :

Timeline

2019-08-08

  • 00:30 UTC - CPU usage on file-33,34,35 starts going up (customer started importing repos with many tags)
  • 01:47 UTC - cache_vulnerability_history feature flag enabled
  • 04:24 UTC - gitaly latency APDEX starts going down, CPU usage on file-33,34,35 reaches 80%
  • 05:10 UTC - web_hook queue starts growing
  • 05:29 UTC - gitaly latency APDEX alert in #alerts-general
  • 06:28 UTC - Incident opened by EOC
  • 06:31 UTC - #backend and #g_gitaly pinged for support
  • 06:41 UTC - IMOC pinged via /pd-mgr
  • 07:21 UTC - status.io post
  • 08:03 UTC - users librideploy and dann30 blocked by abuse team - but it didn't help
  • 08:10 UTC - cache_vulnerability_history feature flag disabled
  • 08:16 UTC - gitaly on file-33 restarted, post receive queue starting to go down a bit, but then rises again
  • 08:25 UTC - gitaly on file-34 restarted
  • 08:29 UTC - gitaly on file-35 restarted
  • 08:50 UTC - collecting profiling data from gitaly on file-33
  • 09:30 UTC - profiling data reveals Gitaly spending a lot of time in FindAllTags
  • 11:08 UTC - starting to deploy hot patch for Gitaly n+1 calls to prod
  • 11:14 UTC - post_receive queue starting to decline
  • 11:20 UTC - deployment of hot patch finished
  • 11:45 UTC - problematic jobs manually killed
  • 13:00 UTC - reports of MRs being closed inadvertently (#1040 (closed))
  • 13:16 UTC - hot patch reverted as it wasn't clear if it's causing the closed MRs
  • 14:30 UTC - manually added web_hook to queue groups on the pages sidekiq nodes to accelerate queue processing
  • 14:55 UTC - queues down to zero
Edited by Henri Philipps