HTTP500's during CI job artifact uploads
Summary
HTTP500's during CI job artifact uploads
Timeline
All times UTC.
2020-04-30
- 16:20 - Incident declared from Slack
- 16:24 - A recent deploy to http://GitLab.com is causing 500 errors to be returned on artifact uploads through CI jobs
- 16:35 - Testing a hot patch on staging to resolve the issue before deploying that patch to production
- 16:48 - Testing of the hot patch on our staging and canary environments is ongoing
- 17:04 - The overall number of artifact upload errors on http://GitLab.com has decreased
- 17:12 - Reverted a previous MR which blocked patch deployment in staging environment: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3293
Details
Example failure: https://gitlab.com/gitlab-org/gitlab/-/jobs/531945088
This is due to a bad query discovered here: gitlab-org/gitlab#16267 (comment 333877045)
Source
Incident declared by jskarbek in Slack via /incident declare
command.
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Edited by Nels Nelson