Cannot upload conan packages due to 411 Length Required from MinIO
Summary
When trying to upload conan packages to the built-in registry, the client fails with 'Internal Server Error'. After tracing this through the logs, it seems GitLab Workhorse triggers this because the built-in MinIO backend returns 411 Length Required when trying to upload the package. I posted this issue here and not in the gitlab rails repo because it seems like a misconfiguration related to MinIO/Workhorse.
Steps to Reproduce:
- Install GitLab with Helm
- Activate License for Premium or Ultimate (I am using the Trial right now, but plan to purchase once it is over)
- Create a New Project
- Create a PAT
- Create a Conan Package Locally
- Run
conan upload
Configuration Used (Helm Values File):
global:
hosts:
domain: REDACTED
gitlab:
name: REDACTED
minio:
name: REDACTED
registry:
name: REDACTED
ssh: REDACTED
ingress:
configureCertmanager: false
class: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
tls:
enabled: true
appConfig:
backups:
bucket: gitlab-main
tmpBucket: gitlab-temp
incomingEmail:
address: 'REDACTED'
enabled: true
host: 'REDACTED'
mailbox: 'Inbox'
password:
secret: git-email
key: imap-password
port: 993
ssl: true
startTls: false
user: 'REDACTED'
expungeDeleted: true
defaultProjectsFeatures:
containerRegistry: false
wiki: false
mergeRequests: false
issues: false
snippets: false
builds: false
cron_jobs:
stuck_ci_jobs_worker:
cron: "0 * * * *"
repository_check_worker:
cron: "20 * * * *"
admin_email_worker:
cron: "0 0 * * 0"
omniauth:
enabled: false
email:
display_name: 'REDACTED'
from: 'REDACTED'
reply_to: 'REDACTED'
smtp:
address: REDACTED
enabled: true
password:
secret: git-email
key: smtp-password
port: 587
starttls_auto: true
user_name: 'REDACTED'
shell:
port: 22
nginx-ingress:
enabled: false
certmanager:
install: false
prometheus:
install: false
gitlab-runner:
runners:
image: 'ubuntu:focal'
imagePullSecrets:
- regcred
registry:
ingress:
tls:
secretName: gitlab-tls-registry
minio:
ingress:
tls:
secretName: gitlab-tls-minio
persistence:
size: 128Gi
imageTag: RELEASE.2020-04-15T19-42-18Z
gitlab:
task-runner:
backups:
cron:
enabled: true
schedule: "30 1 * * *"
objectStorage:
config:
secret: gitlab-backup-storage
key: config
gitaly:
persistence:
size: 256Gi
gitlab-shell:
service:
type: NodePort
webservice:
ingress:
tls:
secretName: gitlab-tls-webservice
Current Behavior:
Any attempt to upload a conan package is met with a 'Internal Server Error' message and doesn't work.
Expected Behavior:
The upload succeeds successfully. I set up a test instance of JFrog Artifactory just to make sure my package wasn't the cause, but the upload works with that, so it seems to be GitLab's issue.
Versions:
Chart: 3.3.3
Platform: Self-Hosted: Kubespray
Kubernetes: Server: 1.17.5 Client: 1.17.4
Helm: Client: v3.2.0
Relevant Logs:
- (From Conan Client):
Uploading to remote 'gitlab':
Uploading REDACTED to remote 'gitlab'
ERROR:
Error uploading file: conanfile.py, 'Internal server error
'
ERROR:
Error uploading file: conanmanifest.txt, 'Internal server error
'
ERROR: REDACTED: Upload recipe to 'gitlab' failed: Execute upload again to retry upload the failed files: conanfile.py, conanmanifest.txt. [Remote: gitlab]
ERROR: Errors uploading some packages
- (From Rails):
2020-05-13T17:12:03.174550815Z *** /var/log/gitlab/production.log ***
2020-05-13T17:12:03.174558608Z Started PUT "/api/v4/packages/conan/v1/files/REDACTED/REDACTED/REDACTED/stable/0/export/conanmanifest.txt/authorize" for REDACTED at 2020-05-13 17:12:02 +0000
2020-05-13T17:12:03.174565808Z
2020-05-13T17:12:03.174571719Z *** /var/log/gitlab/api_json.log ***
2020-05-13T17:12:03.174578799Z {"time":"2020-05-13T17:12:02.388Z","severity":"INFO","duration_s":0.09,"db_duration_s":0.01,"view_duration_s":0.07,"status":200,"method":"PUT","path":"/api/v4/packages/conan/v1/files/REDACTED/REDACTED/REDACTED/stable/0/export/conanmanifest.txt/authorize","params":[],"host":"REDACTED","remote_ip":"REDACTED","ua":"Conan/1.25.0 (Python 3.6.9) python-requests/2.23.0","route":"/api/:version/packages/conan/v1/files/:package_name/:package_version/:package_username/:package_channel/:recipe_revision/export/:file_name/authorize","user_id":REDACTED,"username":"REDACTED","redis_calls":1,"redis_duration_s":0.0,"correlation_id":"REDACTED"}
2020-05-13T17:12:12.175755698Z
- (From Workhorse):
2020-05-13T17:12:02.786031017Z time="2020-05-13T17:12:02Z" level=error msg=error correlation_id=REDACTED error="BodyUploader: upload failed: PUT request http://gitlab-minio-svc:9000/gitlab-packages/tmp/uploads/REDACTED?X-Amz-Expires=15300&X-Amz-Date=20200513T171202Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=REDACTED&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED] returned: 411 Length Required" method=PUT uri=/api/v4/packages/conan/v1/files/REDACTED/REDACTED/REDACTED/stable/0/export/conanmanifest.txt
2020-05-13T17:12:02.7860531Z REDACTED 10.233.90.201 - - [2020/05/13:17:12:02 +0000] "PUT /api/v4/packages/conan/v1/files/REDACTED/REDACTED/REDACTED/stable/0/export/conanmanifest.txt HTTP/1.1" 500 22 "" "Conan/1.25.0 (Python 3.6.9) python-requests/2.23.0" 130
- For some reason, MinIO's logs do not contain the error.
Things I have Tried:
- Updating the MinIO image to the latest (as shown in YAML). This is because after googling around about the 411, I found that there was a bug in MinIO related to this. This doesn't fix it.
- Changing
aws_signature_version
from 4 to 2 in the rails config.yml. This is also from googling as some people get 411 on non-Amazon S3 providers due to the newer signature. This also doesn't fix it.
PS: Not related to the issue, but thanks for all the great work! GitLab has held up to its promise of being a single app for all devops (except this bug) and the Helm chart makes it super convenient to run GitLab.