Make S3 Multipart Upload MaxParts Limit and PartSize Configurable in Workhorse

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

When uploading large files (e.g., >65GB) via GitLab (LFS, CI artifacts, etc.) to S3-compatible storage, uploads fail with the error:
TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000)

This occurs because ​GitLab Workhorse hardcodes two critical values:

  1. ​**MaxUploadParts**: A hardcoded limit of 10,000 parts (source).
  2. ​**DefaultUploadPartSize**: Default part size used for multipart calculations (often defaulting to 5-15MB).

While S3 allows increasing MaxUploadParts (e.g., to 20,000+), ​Workhorse’s internal check rejects the upload before it reaches S3, rendering S3-side configurations ineffective.

​Impact:​​

  • ​Breaks large file uploads: Files >~50GB (with default part sizes) cannot be uploaded.
  • ​Scalability issue: Hinders use cases requiring very large files (ML datasets, media assets, backups).

​Proposed Solution:​​
Make both parameters ​configurable in GitLab:

  1. ​**maxUploadParts**: Add a new setting (e.g., object_store['multipart_upload_max_parts']) to override the default 10,000 limit.
  2. ​**part_size: ​Already exists​ (multipart_upload_part_size), but should be ​documented prominently**​ for large uploads.

​Why this matters:​​

  • Aligns with S3’s configurable MaxUploadParts.
  • Future-proofs GitLab for increasingly common large-data workflows.
  • Avoids silent failures where S3 settings appear sufficient but are bypassed by Workhorse.

​Steps to Reproduce:​​

  1. Configure GitLab with S3 storage.
  2. Attempt to upload a file >65GB (e.g., via LFS).
  3. Observe TotalPartsExceeded error despite S3 allowing >10,000 parts.

​GitLab Version:​​
[Specify your version, e.g., 16.11.0]

​References:​​

  • Error observed: "exceeded total allowed configured MaxUploadParts (10000)"
  • AWS S3 MaxUploadParts docs: link

Key points this issue covers:

  1. ​Problem Clarity: Identifies the hardcoded maxUploadsParts as the root cause.
  2. ​Impact: Explains real-world limitations (65GB files failing).
  3. ​Solution Focus: Requests configurability, not just a higher default.
  4. ​Credibility: References Workhorse code and S3 best practices.
  5. ​Reproducibility: Clear steps for the GitLab team to validate.

Where to submit:

  • ​GitLab Workhorse Repository: gitlab-org/gitlab-workhorse
  • ​GitLab Main Repository​ (if Workhorse changes require coordination): gitlab-org/gitlab

Pro tip:

  • ​Include logs: Attach your error log snippet showing UploadId and the 10000 error.
  • ​Mention your use case: E.g., “We store large AI datasets via GitLab LFS and hit this weekly.”
  • ​Link related issues: Search for similar reports and reference them.
Edited by 🤖 GitLab Bot 🤖