Cannot upload conan packages due to 411 Length Required from MinIO

Summary

When trying to upload conan packages to the built-in registry, the client fails with 'Internal Server Error'. After tracing this through the logs, it seems GitLab Workhorse triggers this because the built-in MinIO backend returns 411 Length Required when trying to upload the package. I posted this issue here and not in the gitlab rails repo because it seems like a misconfiguration related to MinIO/Workhorse.

Steps to Reproduce:

• Install GitLab with Helm
• Activate License for Premium or Ultimate (I am using the Trial right now, but plan to purchase once it is over)
• Create a New Project
• Create a PAT
• Create a Conan Package Locally
• Run conan upload

Configuration Used (Helm Values File):

global:
  hosts:
    domain: REDACTED
    gitlab:
      name: REDACTED
    minio:
      name: REDACTED
    registry:
      name: REDACTED
    ssh: REDACTED
  ingress:
    configureCertmanager: false
    class: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    tls:
      enabled: true
  appConfig:
    backups:
      bucket: gitlab-main
      tmpBucket: gitlab-temp
    incomingEmail:
      address: 'REDACTED'
      enabled: true
      host: 'REDACTED'
      mailbox: 'Inbox'
      password:
        secret: git-email
        key: imap-password
      port: 993
      ssl: true
      startTls: false
      user: 'REDACTED'
      expungeDeleted: true
    defaultProjectsFeatures:
      containerRegistry: false
      wiki: false
      mergeRequests: false
      issues: false
      snippets: false
      builds: false
    cron_jobs:
      stuck_ci_jobs_worker:
        cron: "0 * * * *"
      repository_check_worker:
        cron: "20 * * * *"
      admin_email_worker:
        cron: "0 0 * * 0"
    omniauth:
      enabled: false
  email:
    display_name: 'REDACTED'
    from: 'REDACTED'
    reply_to: 'REDACTED'
  smtp:
    address: REDACTED
    enabled: true
    password:
      secret: git-email
      key: smtp-password
    port: 587
    starttls_auto: true
    user_name: 'REDACTED'
  shell:
    port: 22

nginx-ingress:
  enabled: false

certmanager:
  install: false

prometheus:
  install: false

gitlab-runner:
  runners:
    image: 'ubuntu:focal'
    imagePullSecrets:
      - regcred

registry:
  ingress:
    tls:
      secretName: gitlab-tls-registry

minio:
  ingress:
    tls:
      secretName: gitlab-tls-minio
  persistence:
    size: 128Gi
  imageTag: RELEASE.2020-04-15T19-42-18Z

gitlab:
  task-runner:
    backups:
      cron:
        enabled: true
        schedule: "30 1 * * *"
      objectStorage:
        config:
          secret: gitlab-backup-storage
          key: config
  gitaly:
    persistence:
      size: 256Gi
  gitlab-shell:
    service:
      type: NodePort
  webservice:
    ingress:
      tls:
        secretName: gitlab-tls-webservice

Current Behavior:

Any attempt to upload a conan package is met with a 'Internal Server Error' message and doesn't work.

Expected Behavior:

The upload succeeds successfully. I set up a test instance of JFrog Artifactory just to make sure my package wasn't the cause, but the upload works with that, so it seems to be GitLab's issue.

Versions:

Chart: 3.3.3

Platform: Self-Hosted: Kubespray

Kubernetes: Server: 1.17.5 Client: 1.17.4

Helm: Client: v3.2.0

Relevant Logs:

• (From Conan Client):

Uploading to remote 'gitlab':
Uploading REDACTED to remote 'gitlab'                               
ERROR: 
Error uploading file: conanfile.py, 'Internal server error
'
ERROR: 
Error uploading file: conanmanifest.txt, 'Internal server error
'
ERROR: REDACTED: Upload recipe to 'gitlab' failed: Execute upload again to retry upload the failed files: conanfile.py, conanmanifest.txt. [Remote: gitlab]

ERROR: Errors uploading some packages

• (From Rails):

2020-05-13T17:12:03.174550815Z *** /var/log/gitlab/production.log ***
2020-05-13T17:12:03.174558608Z Started PUT "/api/v4/packages/conan/v1/files/REDACTED/REDACTED/REDACTED/stable/0/export/conanmanifest.txt/authorize" for REDACTED at 2020-05-13 17:12:02 +0000
2020-05-13T17:12:03.174565808Z 
2020-05-13T17:12:03.174571719Z *** /var/log/gitlab/api_json.log ***
2020-05-13T17:12:03.174578799Z {"time":"2020-05-13T17:12:02.388Z","severity":"INFO","duration_s":0.09,"db_duration_s":0.01,"view_duration_s":0.07,"status":200,"method":"PUT","path":"/api/v4/packages/conan/v1/files/REDACTED/REDACTED/REDACTED/stable/0/export/conanmanifest.txt/authorize","params":[],"host":"REDACTED","remote_ip":"REDACTED","ua":"Conan/1.25.0 (Python 3.6.9) python-requests/2.23.0","route":"/api/:version/packages/conan/v1/files/:package_name/:package_version/:package_username/:package_channel/:recipe_revision/export/:file_name/authorize","user_id":REDACTED,"username":"REDACTED","redis_calls":1,"redis_duration_s":0.0,"correlation_id":"REDACTED"}
2020-05-13T17:12:12.175755698Z 

• (From Workhorse):

2020-05-13T17:12:02.786031017Z time="2020-05-13T17:12:02Z" level=error msg=error correlation_id=REDACTED error="BodyUploader: upload failed: PUT request http://gitlab-minio-svc:9000/gitlab-packages/tmp/uploads/REDACTED?X-Amz-Expires=15300&X-Amz-Date=20200513T171202Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=REDACTED&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED] returned: 411 Length Required" method=PUT uri=/api/v4/packages/conan/v1/files/REDACTED/REDACTED/REDACTED/stable/0/export/conanmanifest.txt
2020-05-13T17:12:02.7860531Z REDACTED 10.233.90.201 - - [2020/05/13:17:12:02 +0000] "PUT /api/v4/packages/conan/v1/files/REDACTED/REDACTED/REDACTED/stable/0/export/conanmanifest.txt HTTP/1.1" 500 22 "" "Conan/1.25.0 (Python 3.6.9) python-requests/2.23.0" 130

• For some reason, MinIO's logs do not contain the error.

Things I have Tried:

• Updating the MinIO image to the latest (as shown in YAML). This is because after googling around about the 411, I found that there was a bug in MinIO related to this. This doesn't fix it.
• Changing aws_signature_version from 4 to 2 in the rails config.yml. This is also from googling as some people get 411 on non-Amazon S3 providers due to the newer signature. This also doesn't fix it.

PS: Not related to the issue, but thanks for all the great work! GitLab has held up to its promise of being a single app for all devops (except this bug) and the Helm chart makes it super convenient to run GitLab.