2020-03-12 customers.gitlab.com is down
Summary
More information will be added as we investigate the issue.
Timeline
All times UTC.
2020-03-12
- xx:xx - permissions on the file change
- 11:34 - chef-client run fails due to permissions
- 11:38 - customers.gitlab.com is down
- 11:40 - EOC is paged
- 11:50 - permissions on the file are fixed, unicorn starts successfully
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Workaround
- Look at the files in
/home/gitlab-customers/customers-gitlab-com
for any that aren't owned bygitlab-customers:gitlab-customers
withls -la /home/gitlab-customers/customers-gitlab-com
- Change the ownership of those files, for example:
sudo chown gitlab-customers:gitlab-customers /home/gitlab-customers/customers-gitlab-com/log/production_json.log
Edited by Lyle Kozloff