Change in behavior of /uploads api endpoint resulting in errors
Summary
Uploads to <project_path>/uploads
are failing for api requests which used to work fine.
Steps to reproduce
Retry the release
job in the failing pipeline: https://gitlab.com/gitlab-org/pubsubbeat/-/jobs/1140160903 (I didn't have time yet to boil it down to a curl request).
Example Project
https://gitlab.com/gitlab-org/pubsubbeat
example failing pipelines:
- https://gitlab.com/gitlab-org/pubsubbeat/-/pipelines/278818789
- https://gitlab.com/gitlab-org/pubsubbeat/-/pipelines/278758962
- https://gitlab.com/gitlab-org/pubsubbeat/-/pipelines/278369406
Workhorse errors for this project: https://log.gprd.gitlab.net/goto/bd5f3c6f377dc5b55c00b409dcae13b3
More projects are affected, see another Kibana search below
What is the current bug behavior?
uploads are failing
What is the expected correct behavior?
uploads are working
Relevant logs and/or screenshots
Here's a Kibana search which returns corresponding workhorse errors: https://log.gprd.gitlab.net/goto/b05cf1b878389a509c1bd0c88f2af40d
I suspect the regression was introduced by a release on March 25th, but that's just based on the above search.
Some of these errors are failing for what looks like genuinely wrong API requests. However, many of them are generated by requests that used to be valid. One example of such requests, are those generated by goreleaser
.
Detailed workhorse error message:
handleFileUploads: extract files from multipart: illegal filename: <redacted>
Output of checks
This bug happens on GitLab.com
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
Some suspects:
- https://gitlab.com/gitlab-org/gitlab/-/blob/5eabe70dadec070558b4ec282dbb209295da63d8/workhorse/internal/upload/rewrite.go#L118
- !57250 (merged)
cc @stanhu @nick.thomas I didn't see any hard evidence yet that it's related to your change, it's just based on time of regression and scope of change