[Meta] Failures during artifact uploads on gitlab.com
Summary
We've been experiencing frequent failures on gitlab.com with stuck jobs and artifacts failing to upload. This has led to a frequently broken master:broken, or at least the illusion of one.
Steps to reproduce
Example Project
https://gitlab.com/gitlab-org/gitlab
What is the current bug behavior?
Artifacts fail to upload during assorted CI jobs
What is the expected correct behavior?
Artifacts should upload at the end of CI job execution
Relevant logs and/or screenshots
ERROR: Uploading artifacts to coordinator... error error=couldn't execute POST against https://gitlab.com/api/v4/jobs/352193508/artifacts?artifact_format=gzip&artifact_type=junit&expire_in=31d: Post https://gitlab.com/api/v4/jobs/352193508/artifacts?artifact_format=gzip&artifact_type=junit&expire_in=31d: EOF id=352193508 token=NzY8KVLy
2241 WARNING: Retrying... error=invalid argument
2242 WARNING: Uploading artifacts to coordinator... failed id=352193508 responseStatus=500 Internal Server Error status=500 Internal Server Error token=NzY8KVLy
2243 WARNING: Retrying... error=invalid argument
2244 WARNING: Uploading artifacts to coordinator... failed id=352193508 responseStatus=500 Internal Server Error status=500 Internal Server Error token=NzY8KVLy
2245 FATAL: invalid argument
2247 ERROR: Job failed: exit code 1
https://gitlab.com/gitlab-org/gitlab/-/jobs/352193508
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)