Apply the maximum package file size to every package upload
What does this MR do and why?
With the packages_workhorse_maximum_size_for_all_uploads feature flag on, Workhorse refuses a Conan, Debian, generic, ML model or Terraform module package file over the maximum file size of its format before it stores the file, and answers with the same 400 File is too large JSON that Rails sends today. These formats send a length, and for those uploads authorize_workhorse! left MaximumSize out of the authorize response, so Workhorse stored the whole file and Rails refused it afterwards. Maven, npm, Cargo, NuGet, PyPI, RubyGems and Helm already send the limit and get the Workhorse 413.
Two parts:
- Workhorse: the authorize response can carry
MaximumSizeResponse(StatusCode,ContentType,Body). When an upload is overMaximumSize, the request body uploader answers with it instead of the default413text. - Rails:
authorize_workhorse!sendsmaximum_sizefor uploads with a length, together with the body thatbad_request!renders.ObjectStorage::Concern.workhorse_authorizetakes amaximum_size_response:keyword for it.
The flag is gitlab_com_derisk, default off, with the project as actor; rollback is to disable the flag. Enable it only after the Workhorse change is deployed: an older Workhorse ignores the field and answers 413.
Design notes
- Why the condition existed:
params[:maximum_size] = maximum_size unless has_lengthcame from 6a9e9f29 in !23599 (merged) (January 2020, NuGet multipart uploads). At that timemaximum_sizeonly went toObjectStorage::DirectUpload, which needs it to size the presigned AWS multipart parts, and only for uploads without a length. The top-levelMaximumSizethat Workhorse uses as a body limit came later, in 5ce80ffc (July 2020), and the helper condition was not revisited. - Why not refuse in Rails from a forwarded
Content-Length(an earlier revision of this MR): a client can craft request headers, and a chunked upload has no length, so Workhorse stored the file before Rails refused it (!265805 (comment 4117629452)). - Workhorse applies
MaximumSizeto every upload (setupSizeLimiterinworkhorse/internal/upload/destination/destination.go): with a known length over the limit it fails before it reads the body; otherwise a hard-limit reader stops at the limit. The response was the only fixed part, andMaximumSizeResponsemakes it configurable. - Bound: Workhorse refuses at
> MaximumSize,PlanLimits#exceeded?at>=. A file of exactly the limit is stored and gets the Rails400after the upload, as today. A limit of0is unlimited in both. - The canned body is rendered with the Grape error formatter and content type of the endpoint's format, so Debian (
format :txt) getstext/plainand Conan gets theerrorsarray thatConan::ApiErrorFormatteradds through theapi_error_bodyhook. A client gets one body whichever side refuses the file. - Formats without a length keep the
413: nothing sendsMaximumSizeResponsefor them, and the multipart handler does not read it. - With
has_length: true,ObjectStorage::DirectUploaddoes not use multipart uploads, so the extramaximum_sizechanges only the top-levelMaximumSize. - Generic and Terraform module uploads use
use_final_store_path: true. Today a file that Rails refuses stays in object storage untilObjectStorage::DeleteStaleDirectUploadsServiceremoves it after 3 hours. With the flag on, Workhorse refuses a file over the limit before it stores it. - Specs:
TestRequestBodyMaximumSizeResponsecovers the Workhorse response. The helper spec covers both flag states, the custom message and uploads without a length. Theworkhorse_authorizespec covers the new keyword. A request spec per format checks that the authorize response carries the plan limit asMaximumSizeand the400JSON asMaximumSizeResponsewith the flag on, and neither with the flag off.
References
- Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/635054
- Feature flag rollout: https://gitlab.com/gitlab-org/gitlab/-/work_items/636673
- Code owners: the Package Registry group (
@gitlab-org/ci-cd/package-stage/package-registry-group) and Workhorse maintainers.
Screenshots or screen recordings
No UI change.
How to set up and validate locally
-
Rebuild Workhorse:
cd workhorse && make && gdk restart workhorse. -
In a Rails console, lower the generic package limit:
Plan.default.actual_limits.update!(generic_packages_max_file_size: 1.megabyte). -
Create a personal access token with the
apiscope for a user with the Developer role on a project. SetTOKEN=<your_access_token>andPROJECT_ID=<project_id>. -
Create a 2 MB file:
head -c 2097152 /dev/zero > big.bin. -
Upload it:
curl --verbose --request PUT --header "PRIVATE-TOKEN: $TOKEN" --upload-file big.bin \ "http://gdk.test:3000/api/v4/projects/$PROJECT_ID/packages/generic/my_package/0.0.1/big.bin"Expect (flag off):
{"message":"400 Bad request - File is too large"}after curl sends the file. Workhorse stored the file before Rails refused it. -
Enable the flag:
Feature.enable(:packages_workhorse_maximum_size_for_all_uploads). -
Upload
big.binagain. Expect the same400body, with noHTTP/1.1 100 Continuebefore it: Workhorse refuses the file before it reads it. The Workhorse log hasentity is too large. -
Upload it chunked: add
--header "Transfer-Encoding: chunked". Expect the same400body, and no new object in MinIO. -
Upload a smaller file (
head -c 524288 /dev/zero > small.bin). Expect: status201. -
Restore the limit:
Plan.default.actual_limits.update!(generic_packages_max_file_size: 5.gigabytes).
More scenarios
- All five formats go through the Workhorse
requestBodyUploader(workhorse/internal/upstream/routes.go). - ML model, Conan, Debian and Terraform module uploads behave the same with their own plan limits. The ML model message is
Artifact file is too large. - Formats without a length (NuGet, PyPI, RubyGems, Helm) and the formats that call
PackageFileUploader.workhorse_authorizedirectly (Maven, npm, Cargo) are unchanged.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.