new_note sidekiq queue is growing
Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.
Summary
A user generated many notes on a single commit which slowed down the new_note
sidekiq queue and led to a delay of send out notifications.
new_note sidekiq jobs dropped: (https://log.gitlab.net/goto/a768cc64e673b474eb59bce0dac36a57)
Service(s) affected : ~"Service:Sidekiq" Team attribution : Minutes downtime or degradation : 220
Timeline
2019-08-05
- 11:16 UTC - 1410 note request to one commit added until 11:21 UTC
- 11:16 UTC -
new_note
queue starts to grow - 12:24 UTC - Pagerduty Alert "Large amount of new_note sidekiq queued jobs: 6541"
- 12:37 UTC - Incident started by EOC
- 12:42 UTC - IMOC pinged by EOC
- 12:49 UTC - EOC asking for help in backend Slack channel
- 12:52 UTC - IMOC responded
- 12:55 UTC - IMOC pinging Andrew
- 13:00 UTC - Andrew joining Incident Room
- 13:07 UTC - response from backend engineer in Slack
- 13:17 UTC - status.io incident and tweet
- 13:46 UTC - commit causing the issue identified
- 13:51 UTC - status.io update (tweet)
- 14:15 UTC - rebooting console which became unresponsive due to profiling
- 14:18 UTC - manually purging the queue, removing the notes caused by the identified commit
- 14:35 UTC - status.io update
- 14:56 UTC - all pending jobs have been processed
- 15:03 UTC - status.io incident resolved
Edited by 🤖 GitLab Bot 🤖