2020-05-04: mail queues building up
Summary
2020-05-04: mail queues building up
Timeline
All times UTC.
2020-05-04
-
14:50
- @jarv posts a message in Slack#production
with a screenshot showing a growing mail queue. -
14:53
- Incident declared from Slack. -
15:08
- @craigf opens MR to double the number of nodes processing thelow-urgency-cpu-bound
queue from 2 to 4. -
15:23
- Request for assistance made by @AnthonySandoval to @sarcila in#dev-escalation
. -
15:27
- @sarcila joins the Situation Room zoom meeting. -
15:27
- The queue hits it's max size of 22k. -
15:42
- @craigf opens MR to double the number of nodes processinglow-urgency-cpu-bound
queue from 4 to 8. -
16:06
- The queue begins to drop from 20k. -
16:09
- The queue is empty and drops back to 0. -
16:11
- @andrewn recommends declaring the incident over. -
16:11
- @nnelson closes this issue–ending the incident.
Details
After being alerted to a mail queues building up, we determined that the low-urgency-cpu-bound
Sidekiq queue was saturated. The change that introduced the increased utilization was gitlab-org/gitlab!30731 (merged).
Source
Incident declared by cfurman in Slack via /incident declare
command.
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Edited by Nels Nelson