Apply the maximum package file size to every package upload

What does this MR do and why?

With the packages_workhorse_maximum_size_for_all_uploads feature flag on, Workhorse refuses a Conan, Debian, generic, ML model or Terraform module package file over the maximum file size of its format before it stores the file, and answers with the same 400 File is too large JSON that Rails sends today. These formats send a length, and for those uploads authorize_workhorse! left MaximumSize out of the authorize response, so Workhorse stored the whole file and Rails refused it afterwards. Maven, npm, Cargo, NuGet, PyPI, RubyGems and Helm already send the limit and get the Workhorse 413.

Two parts:

  • Workhorse: the authorize response can carry MaximumSizeResponse (StatusCode, ContentType, Body). When an upload is over MaximumSize, the request body uploader answers with it instead of the default 413 text.
  • Rails: authorize_workhorse! sends maximum_size for uploads with a length, together with the body that bad_request! renders. ObjectStorage::Concern.workhorse_authorize takes a maximum_size_response: keyword for it.

The flag is gitlab_com_derisk, default off, with the project as actor; rollback is to disable the flag. Enable it only after the Workhorse change is deployed: an older Workhorse ignores the field and answers 413.

Design notes
  • Why the condition existed: params[:maximum_size] = maximum_size unless has_length came from 6a9e9f29 in !23599 (merged) (January 2020, NuGet multipart uploads). At that time maximum_size only went to ObjectStorage::DirectUpload, which needs it to size the presigned AWS multipart parts, and only for uploads without a length. The top-level MaximumSize that Workhorse uses as a body limit came later, in 5ce80ffc (July 2020), and the helper condition was not revisited.
  • Why not refuse in Rails from a forwarded Content-Length (an earlier revision of this MR): a client can craft request headers, and a chunked upload has no length, so Workhorse stored the file before Rails refused it (!265805 (comment 4117629452)).
  • Workhorse applies MaximumSize to every upload (setupSizeLimiter in workhorse/internal/upload/destination/destination.go): with a known length over the limit it fails before it reads the body; otherwise a hard-limit reader stops at the limit. The response was the only fixed part, and MaximumSizeResponse makes it configurable.
  • Bound: Workhorse refuses at > MaximumSize, PlanLimits#exceeded? at >=. A file of exactly the limit is stored and gets the Rails 400 after the upload, as today. A limit of 0 is unlimited in both.
  • The canned body is rendered with the Grape error formatter and content type of the endpoint's format, so Debian (format :txt) gets text/plain and Conan gets the errors array that Conan::ApiErrorFormatter adds through the api_error_body hook. A client gets one body whichever side refuses the file.
  • Formats without a length keep the 413: nothing sends MaximumSizeResponse for them, and the multipart handler does not read it.
  • With has_length: true, ObjectStorage::DirectUpload does not use multipart uploads, so the extra maximum_size changes only the top-level MaximumSize.
  • Generic and Terraform module uploads use use_final_store_path: true. Today a file that Rails refuses stays in object storage until ObjectStorage::DeleteStaleDirectUploadsService removes it after 3 hours. With the flag on, Workhorse refuses a file over the limit before it stores it.
  • Specs: TestRequestBodyMaximumSizeResponse covers the Workhorse response. The helper spec covers both flag states, the custom message and uploads without a length. The workhorse_authorize spec covers the new keyword. A request spec per format checks that the authorize response carries the plan limit as MaximumSize and the 400 JSON as MaximumSizeResponse with the flag on, and neither with the flag off.

References

Screenshots or screen recordings

No UI change.

How to set up and validate locally

  1. Rebuild Workhorse: cd workhorse && make && gdk restart workhorse.

  2. In a Rails console, lower the generic package limit: Plan.default.actual_limits.update!(generic_packages_max_file_size: 1.megabyte).

  3. Create a personal access token with the api scope for a user with the Developer role on a project. Set TOKEN=<your_access_token> and PROJECT_ID=<project_id>.

  4. Create a 2 MB file: head -c 2097152 /dev/zero > big.bin.

  5. Upload it:

    curl --verbose --request PUT --header "PRIVATE-TOKEN: $TOKEN" --upload-file big.bin \
      "http://gdk.test:3000/api/v4/projects/$PROJECT_ID/packages/generic/my_package/0.0.1/big.bin"

    Expect (flag off): {"message":"400 Bad request - File is too large"} after curl sends the file. Workhorse stored the file before Rails refused it.

  6. Enable the flag: Feature.enable(:packages_workhorse_maximum_size_for_all_uploads).

  7. Upload big.bin again. Expect the same 400 body, with no HTTP/1.1 100 Continue before it: Workhorse refuses the file before it reads it. The Workhorse log has entity is too large.

  8. Upload it chunked: add --header "Transfer-Encoding: chunked". Expect the same 400 body, and no new object in MinIO.

  9. Upload a smaller file (head -c 524288 /dev/zero > small.bin). Expect: status 201.

  10. Restore the limit: Plan.default.actual_limits.update!(generic_packages_max_file_size: 5.gigabytes).

More scenarios
  • All five formats go through the Workhorse requestBodyUploader (workhorse/internal/upstream/routes.go).
  • ML model, Conan, Debian and Terraform module uploads behave the same with their own plan limits. The ML model message is Artifact file is too large.
  • Formats without a length (NuGet, PyPI, RubyGems, Helm) and the formats that call PackageFileUploader.workhorse_authorize directly (Maven, npm, Cargo) are unchanged.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Eduardo Bonet

Merge request reports

Loading
Loading