2019-07-03: Degraded Redis performance
Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.
Summary
A brief summary of what happened. Try to make it as executive-friendly as possible.
Service(s) affected : Team attribution : Minutes downtime or degradation :
Timeline
2019-07-03
- 09:15 UTC - PING latency to redis-cache-02 is hovering between 6
7ms, which is slightly higher than yesterday (12ms) - 09:55 UTC - Increased
loglevel
toverbose
on redis-cache-02 to see if there is any useful info there - 10:12 UTC - Reverted
loglevel
tonotice
on redis-cache-02; nothing of importance was being logged - 11:25 UTC - We applied https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14509 to production
- 13:52 UTC - We disabled the performance bar
- 08:50 UTC (July 4th) - The performance bare is re-enabled
Edited by John Jarvis