customers.gitlab.com is returing 502 errors

Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.

---

Summary

A brief summary of what happened. Try to make it as executive-friendly as possible.

Service(s) affected : Team attribution : Minutes downtime or degradation :

Timeline

2019-06-11

• 09:38 UTC - this change gets merged: https://gitlab.com/gitlab-org/customers-gitlab-com/commit/e78b2a3ca1356db9d4e5b49f402d955208102c9d
• 10:05 UTC - chef-client run on customers.gitlab.com fails to install healthcheck gem, the website goes down
• 10:38 UTC - on-call gets notified about outage, issues start coming in
• 11:24 UTC - gemfile.lock is manually patched on the machine, the website goes online again