2020-02-04 - GitLab.com Registry Down

Summary

The GitLab container registry is not responding and returning 503s.

Timeline

All times UTC.

2020-02-04

• 20:47 - A dry-run pipeline began to test registry change
• 20:53 - PagerDuty alert went off for registry service connection errors.
• 20:59 - Incident issue created
• 21:01 - Someone familiar with the registry is engaged
• 21:10 - The bad pipeline is identified
• 21:11 - Status page updates
• 21:12 - A previous known good pipeline is ran to apply the correct config
• 21:13 - Pipeline is complete
• 21:14 - Registry manually scaled up to get enough pods quicker than the autoscaler would have
• 21:18 - Registry is verified working