Uploading artifacts fails with 500 error
Gitlab is self-hosted and managed with Helm Chart.
Getting random 500 artifact uploads errors-
- filesizes are rather small, beginning from 10MB.
- Nginx has been set to have unlimited filesizes.
- Containers are running in privileged mode
- No issues for the past 3 months (last upgrade)
WARNING: Uploading artifacts as "archive" to coordinator... failed id=773425 responseStatus=500 Internal Server Error status=500 Internal Server Error token=oxdqvuK2
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed id=773425 responseStatus=500 Internal Server Error status=500 Internal Server Error token=oxdqvuK2
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed id=773425 responseStatus=500 Internal Server Error status=500 Internal Server Error token=oxdqvuK2
Actual behaviour
gitlab-runner starts uploading files, then it silently breaks upload and terminate connection. gitlab shows 500 error coming from workhorse/webservice.
A simplified build has been completed to show that even miniscule filesizes also still fail.
stages:
- build
test:
stage: build
script:
- touch gitversion.properties
- chmod 777 ./gitversion.properties
- ls -ltr
artifacts:
paths:
- ./gitversion.properties
This fails with a 500 error.
this log output from a combination of various workhorse/ingress/webservice
[ldn-gitlab-webservice-69f47898cc-2wzsb gitlab-workhorse] <redacted>.co.uk <redacted> - - [2022/03/24:18:13:44 +0000] "POST <redacted>/gitt-upload-pack HTTP/1.1" 200 76873236 "" "git/2.22.4" 1804
[ldn-gitlab-nginx-ingress-controller-55454b57c6-5x9m7] <redacted> - [<redacted>] - - [24/Mar/2022:18:13:47 +0000] "POST /api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive HTTP/1.1" 500 22 "-" "gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)" 879 0.150 [gitlab-ldn-gitlab-webservice-8181] <redacted>:8181 22 0.152 500 c965f3e169e67d1dd445161d044f2279
[ldn-gitlab-webservice-69f47898cc-gdsw8 webservice] Started POST "/api/v4/jobs/773702/artifacts/authorize?artifact_format=zip&artifact_type=archive" for <redacted> at 2022-03-24 18:13:46 +0000
[ldn-gitlab-webservice-69f47898cc-gdsw8 webservice] {"time":"2022-03-24T18:13:47.061Z","severity":"INFO","duration_s":0.13643,"db_duration_s":0.00326,"view_duration_s":0.13317,"status":200,"method":"POST","path":"/api/v4/jobs/773702/artifacts/authorize","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"}],"host":"<redacted>.co.uk","remote_ip":"<redacted>","ua":"gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)","route":"/api/:version/jobs/:id/artifacts/authorize","redis_calls":1,"redis_duration_s":0.000206,"redis_read_bytes":2,"redis_write_bytes":113,"redis_cache_calls":1,"redis_cache_duration_s":0.000206,"redis_cache_read_bytes":2,"redis_cache_write_bytes":113,"correlation_id":"c965f3e169e67d1dd445161d044f2279","meta.user":"xxx","meta.project":"<redacted>","meta.root_namespace":"<redacted>","meta.caller_id":"/api/:version/jobs/:id/artifacts/authorize"}
[ldn-gitlab-webservice-69f47898cc-gdsw8 gitlab-workhorse] time="2022-03-24T18:13:47Z" level=error msg=error correlation_id=STDo66RvIj7 error="handleFileUploads: extract files from multipart: persisting multipart file: create temporary buffer file: open /tmp/gitlab/part-buffer213205036: no such file or directory" method=POST uri="/api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive"
[ldn-gitlab-webservice-69f47898cc-gdsw8 gitlab-workhorse] <redacted>.co.uk <redacted> - - [2022/03/24:18:13:47 +0000] "POST /api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive HTTP/1.1" 500 22 "" "gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)" 148
[ldn-gitlab-webservice-69f47898cc-2wzsb webservice] Started POST "/api/v4/jobs/773702/artifacts/authorize?artifact_format=zip&artifact_type=archive" for <redacted> at 2022-03-24 18:13:48 +0000
[ldn-gitlab-nginx-ingress-controller-55454b57c6-5x9m7] <redacted> - [<redacted>] - - [24/Mar/2022:18:13:48 +0000] "POST /api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive HTTP/1.1" 500 22 "-" "gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)" 879 0.102 [gitlab-ldn-gitlab-webservice-8181] <redacted>:8181 22 0.104 500 b819d3368bba94ad4f1f0650d59b568e
[ldn-gitlab-webservice-69f47898cc-2wzsb webservice] {"time":"2022-03-24T18:13:48.265Z","severity":"INFO","duration_s":0.08831,"db_duration_s":0.00622,"view_duration_s":0.08209,"status":200,"method":"POST","path":"/api/v4/jobs/773702/artifacts/authorize","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"}],"host":"<redacted>.co.uk","remote_ip":"<redacted>","ua":"gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)","route":"/api/:version/jobs/:id/artifacts/authorize","redis_calls":1,"redis_duration_s":0.000798,"redis_read_bytes":2,"redis_write_bytes":113,"redis_cache_calls":1,"redis_cache_duration_s":0.000798,"redis_cache_read_bytes":2,"redis_cache_write_bytes":113,"correlation_id":"b819d3368bba94ad4f1f0650d59b568e","meta.user":"xxx","meta.project":"<redacted>","meta.root_namespace":"<redacted>","meta.caller_id":"/api/:version/jobs/:id/artifacts/authorize"}
[ldn-gitlab-webservice-69f47898cc-2wzsb gitlab-workhorse] time="2022-03-24T18:13:48Z" level=error msg=error correlation_id=sZPEEiD1dn3 error="handleFileUploads: extract files from multipart: persisting multipart file: create temporary buffer file: open /tmp/gitlab/part-buffer402913395: no such file or directory" method=POST uri="/api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive"
Versions of components
Helm Chart version: 4.1.12
GitLab 13.1.11-ee (db98e78d9d7)
GitLab Shell 13.3.0
GitLab Workhorse v8.35.0
GitLab API v4
Ruby 2.6.6p146
Rails 6.0.3.1
PostgreSQL 11.11
Not sure what to look for now?
Edited by dev ops