Outbound Email from GitLab.com was failing for approximately 30 minutes on GitLab.com
At 18:50:06 UTC emails started to fail to send.
This was a self inflicted outage that was a part of the maintenance plan here: https://gitlab.com/gitlab-com/infrastructure/issues/3507
timeline
- 18:57 - Issue first noticed
- 19:01 - Notification to the public
- 19:07 - Rollback of credentials completed, chef runs targeting the sidekiq besteffort nodes began
- 19:20 - Monitoring the issue showed that it persisted
- 19:25 - Various sidekiq servers were manually HUP’d hoping to clear out the potential lingering processes
- 19:37 - Last failure occurred
- 19:44 - Communication of incident closure to the public
Edited by 🤖 GitLab Bot 🤖