2019-07-03: Degraded Redis performance

Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.

Summary

A brief summary of what happened. Try to make it as executive-friendly as possible.

Service(s) affected : Team attribution : Minutes downtime or degradation :

Timeline

2019-07-03

  • 09:15 UTC - PING latency to redis-cache-02 is hovering between 67ms, which is slightly higher than yesterday (12ms)
  • 09:55 UTC - Increased loglevel to verbose on redis-cache-02 to see if there is any useful info there
  • 10:12 UTC - Reverted loglevel to notice on redis-cache-02; nothing of importance was being logged
  • 11:25 UTC - We applied https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14509 to production
  • 13:52 UTC - We disabled the performance bar
  • 08:50 UTC (July 4th) - The performance bare is re-enabled
Edited by John Jarvis