Uploading artifacts fails with 400 Bad request "file is invalid" error using Helm charts version
Summary
Hi!
I have a Gitlab instance installed from the Helm charts on a k8s cluster, connected to a MinIO for object storage (using Bitnami charts and not the Gitlab included version).
We recently migrated from an Omnibus installation and everything works great except build artifacts upload (job logs work ok). Everytime a CI job tries to upload an artifact, whether it be a single file or a directory, it fails giving a 400 error with no more information.
Here is what the api_json.log shows:
{"time":"2023-11-17T15:15:19.527Z","severity":"INFO","duration_s":0.00465,"db_duration_s":0.00047,"view_duration_s":0.00418,"status":400,"method":"POST","path":"/api/v4/jobs/447046/artifacts","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"},{"key":"file","value":{"filename":"artifacts.zip","type":"application/octet-stream","name":"file","tempfile":null,"head":"Content-Disposition: form-data; name=\"file\"; filename=\"artifacts.zip\"\r\nContent-Type: application/octet-stream\r\n"}}],"host":"[REDACTED]-gitlab-webservice-default.gitlab-server.svc.cluster.local","remote_ip":"100.64.1.102","ua":"gitlab-runner-helper 16.5.0 (16-5-stable; go1.20.10; linux/amd64)","route":"/api/:version/jobs/:id/artifacts","redis_calls":2,"redis_duration_s":0.000482,"redis_write_bytes":114,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000482,"redis_shared_state_write_bytes":114,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":0,"db_ci_count":1,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.010171,"mem_objects":6748,"mem_bytes":607840,"mem_mallocs":2908,"mem_total_bytes":877760,"pid":36,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"0d72fc3c-bec1-43af-8334-2e0f14a23844","meta.caller_id":"POST /api/:version/jobs/:id/artifacts","meta.remote_ip":"100.64.1.102","meta.feature_category":"build_artifacts","meta.user":"l.baillard","meta.user_id":39,"meta.project":"l.baillard/tests-ci","meta.root_namespace":"l.baillard","meta.client_id":"user/39","meta.pipeline_id":48222,"meta.job_id":447046,"content_length":"452","request_urgency":"low","target_duration_s":5}
The object storage configuration has been verified to work on other Gitlab services. The job logs are uploaded correctly.
There is no useful info in the various gitlab logs. I can see the POST request on the webservice and the 400 status but no errors anywhere. There is only a generic error message file is invalid
in the response from the web service when trying to upload the artifacts that I found doing a tcpdump
:
HTTP/1.0 400 Bad Request
Content-Type: application/json
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
vary: Origin
Cache-Control: no-cache
X-Runtime: 0.014332
X-Gitlab-Meta: {"correlation_id":"1b2f0aed-ab35-49d4-aec9-0bac6b3fda19","version":"1"}
X-Request-Id: 1b2f0aed-ab35-49d4-aec9-0bac6b3fda19
Content-Length: 27
{"error":"file is invalid"}
MinIO does not show any more errors. It even seems that the request does not got to the MinIO (can’t see it when showing MinIO traces). I confirmed that by switching artifacts storage to an AWS S3 bucket and the result was the same.
The only info that I found related to this issue, here or in the Gitlab issues, either are way old and do not match my issue or are not related.
I have tried enabling all the debug log options I could find in Gitlab and Gitlab Runner. I have enabled Sentry reporting for Gitlab and the runner. I have tried different CI configurations for the artifacts (reports, plain files, folders, feature flags to enable different Zip archiving methods). Finally, I tried the same configuration (Helm values files for Gitlab, Gitlab Runner and MinIO) on a k3s cluster (so not the k8s where the prod Gitlab instance is) with the same result. None of this yielded any result.
Any help or pointers would be greatly appreciated!
Versions info:
- Gitlab: 16.5.1-ee
- Gitlab charts: 7.5.1
- Gitlab Runner charts: 0.58.1
- Bitnami MinIO charts: 12.8.18
Steps to reproduce
- Deploy a Bitnami MinIO instance on a k3s/k8s cluster
- Deploy the Helm Gitlab charts v7.5.1 on the k3s/k8s cluster
- Deploy the Helm Gitlab Runner charts v0.58.1
- Create a test project on Gitlab with a simple
.gitlab-ci.yml
file creating an artifact from one text file - Run the CI pipeline
Example Project
No example project to mention
What is the current bug behavior?
The job artifacts don't get uploaded and the CI job fails showing a "400 Bad request" error from the coordinator.
What is the expected correct behavior?
Job artifacts successfully uploads to S3 storage.
Relevant logs and/or screenshots
See summary
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
gitlab-rake gitlab:env:info System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.4.20 Bundler Version:2.4.20 Rake Version: 13.0.6 Redis Version: 6.0.16 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 16.5.1-ee Revision: 55da9ccb652 Directory: /srv/gitlab DB Adapter: PostgreSQL DB Version: 13.6 URL: https://gitlab.[REDACTED] HTTP Clone URL: https://gitlab.[REDACTED]/some-group/some-project.git SSH Clone URL: git@gitlab.[REDACTED]:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: no GitLab Shell Version: 14.29.0 Repository storages: - default: tcp://[REDACTED]-gitlab-gitaly-0.[REDACTED]-gitlab-gitaly.gitlab-server.svc:8075 GitLab Shell path: /home/git/gitlab-shell Gitaly - default Address: tcp://[REDACTED]-gitlab-gitaly-0.[REDACTED]-gitlab-gitaly.gitlab-server.svc:8075 - default Version: 16.5.1 - default Git Version: 2.42.0
Results of GitLab application Check
(checks ran on the toolbox pod of the Kubernetes deployment so I expect that some checks might not be accurate)
Expand for output related to the GitLab application check
gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.29.0 ? ... OK (14.29.0) Running /home/git/gitlab-shell/bin/check gitlab-shell self-check failed Try fixing it: Make sure GitLab is running; Check the gitlab-shell configuration file: sudo -u git -H editor /home/git/gitlab-shell/config.yml Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... no Try fixing it: Install the Service For more information see: doc/install/installation.md in section "Install the Service" Please fix the error above and rerun the checks. Systemd unit files or init script up-to-date? ... can't check because of previous errors Projects have namespace: ... 2/2 ... yes 15/13 ... yes 2/15 ... yes 2/16 ... yes 2/17 ... yes 2/21 ... yes 16/24 ... yes 15/25 ... yes 15/26 ... yes 195/27 ... yes 195/29 ... yes 2/30 ... yes 2/31 ... yes 8/33 ... yes 60/34 ... yes 20/37 ... yes 20/38 ... yes 16/40 ... yes 2/41 ... yes 2/42 ... yes 2/43 ... yes 24/46 ... yes 24/47 ... yes 15/48 ... yes 2/49 ... yes 2/50 ... yes 60/52 ... yes 2/53 ... yes 2/54 ... yes 2/55 ... yes 2/57 ... yes 60/60 ... yes 7/61 ... yes 60/62 ... yes 2/63 ... yes 60/64 ... yes 2/65 ... yes 2/66 ... yes 4/67 ... yes 2/72 ... yes 2/73 ... yes 33/76 ... yes 2/78 ... yes 33/79 ... yes 34/80 ... yes 2/81 ... yes 2/82 ... yes 20/84 ... yes 37/85 ... yes 4/86 ... yes 4/88 ... yes 43/89 ... yes 15/90 ... yes 16/91 ... yes 2/94 ... yes 2/95 ... yes 2/96 ... yes 2/97 ... yes 2/99 ... yes 43/100 ... yes 44/102 ... yes 4/104 ... yes 51/105 ... yes 51/106 ... yes 51/107 ... yes 15/108 ... yes 2/110 ... yes 2/111 ... yes 53/112 ... yes 195/113 ... yes 51/114 ... yes 2/116 ... yes 2/117 ... yes 55/118 ... yes 53/119 ... yes 51/120 ... yes 2/121 ... yes 61/122 ... yes 195/123 ... yes 2/124 ... yes 2/125 ... yes 14/126 ... yes 2/127 ... yes 44/128 ... yes 63/129 ... yes 195/130 ... yes 2/133 ... yes 2/136 ... yes 2/137 ... yes 14/138 ... yes 14/140 ... yes 53/141 ... yes 14/142 ... yes 4/143 ... yes 185/144 ... yes 14/145 ... yes 197/148 ... yes 53/149 ... yes 176/150 ... yes 48/151 ... yes 197/153 ... yes 197/154 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 3.0.6 ? ... yes (3.0.6) Git user has default SSH configuration? ... yes Active users: ... 32 Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled) All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
N/A