Skip to content

Uploading artifacts fails with 500 error

Gitlab is self-hosted and managed with Helm Chart.

Getting random 500 artifact uploads errors-

  • filesizes are rather small, beginning from 10MB.
  • Nginx has been set to have unlimited filesizes.
  • Containers are running in privileged mode
  • No issues for the past 3 months (last upgrade)
 WARNING: Uploading artifacts as "archive" to coordinator... failed  id=773425 responseStatus=500 Internal Server Error status=500 Internal Server Error token=oxdqvuK2
 WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
 WARNING: Uploading artifacts as "archive" to coordinator... failed  id=773425 responseStatus=500 Internal Server Error status=500 Internal Server Error token=oxdqvuK2
 WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
 WARNING: Uploading artifacts as "archive" to coordinator... failed  id=773425 responseStatus=500 Internal Server Error status=500 Internal Server Error token=oxdqvuK2

Actual behaviour

gitlab-runner starts uploading files, then it silently breaks upload and terminate connection. gitlab shows 500 error coming from workhorse/webservice.

A simplified build has been completed to show that even miniscule filesizes also still fail.

stages:
  - build

test:
  stage: build
  script:
    - touch gitversion.properties
    - chmod 777 ./gitversion.properties
    - ls -ltr 
  artifacts:
    paths:
      - ./gitversion.properties

This fails with a 500 error.

this log output from a combination of various workhorse/ingress/webservice

[ldn-gitlab-webservice-69f47898cc-2wzsb gitlab-workhorse] <redacted>.co.uk <redacted> - - [2022/03/24:18:13:44 +0000] "POST <redacted>/gitt-upload-pack HTTP/1.1" 200 76873236 "" "git/2.22.4" 1804 
[ldn-gitlab-nginx-ingress-controller-55454b57c6-5x9m7] <redacted> - [<redacted>] - - [24/Mar/2022:18:13:47 +0000] "POST /api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive HTTP/1.1" 500 22 "-" "gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)" 879 0.150 [gitlab-ldn-gitlab-webservice-8181] <redacted>:8181 22 0.152 500 c965f3e169e67d1dd445161d044f2279 
[ldn-gitlab-webservice-69f47898cc-gdsw8 webservice] Started POST "/api/v4/jobs/773702/artifacts/authorize?artifact_format=zip&artifact_type=archive" for <redacted> at 2022-03-24 18:13:46 +0000 
[ldn-gitlab-webservice-69f47898cc-gdsw8 webservice] {"time":"2022-03-24T18:13:47.061Z","severity":"INFO","duration_s":0.13643,"db_duration_s":0.00326,"view_duration_s":0.13317,"status":200,"method":"POST","path":"/api/v4/jobs/773702/artifacts/authorize","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"}],"host":"<redacted>.co.uk","remote_ip":"<redacted>","ua":"gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)","route":"/api/:version/jobs/:id/artifacts/authorize","redis_calls":1,"redis_duration_s":0.000206,"redis_read_bytes":2,"redis_write_bytes":113,"redis_cache_calls":1,"redis_cache_duration_s":0.000206,"redis_cache_read_bytes":2,"redis_cache_write_bytes":113,"correlation_id":"c965f3e169e67d1dd445161d044f2279","meta.user":"xxx","meta.project":"<redacted>","meta.root_namespace":"<redacted>","meta.caller_id":"/api/:version/jobs/:id/artifacts/authorize"} 
[ldn-gitlab-webservice-69f47898cc-gdsw8 gitlab-workhorse] time="2022-03-24T18:13:47Z" level=error msg=error correlation_id=STDo66RvIj7 error="handleFileUploads: extract files from multipart: persisting multipart file: create temporary buffer file: open /tmp/gitlab/part-buffer213205036: no such file or directory" method=POST uri="/api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive" 
[ldn-gitlab-webservice-69f47898cc-gdsw8 gitlab-workhorse] <redacted>.co.uk <redacted> - - [2022/03/24:18:13:47 +0000] "POST /api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive HTTP/1.1" 500 22 "" "gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)" 148 
[ldn-gitlab-webservice-69f47898cc-2wzsb webservice] Started POST "/api/v4/jobs/773702/artifacts/authorize?artifact_format=zip&artifact_type=archive" for <redacted> at 2022-03-24 18:13:48 +0000 
[ldn-gitlab-nginx-ingress-controller-55454b57c6-5x9m7] <redacted> - [<redacted>] - - [24/Mar/2022:18:13:48 +0000] "POST /api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive HTTP/1.1" 500 22 "-" "gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)" 879 0.102 [gitlab-ldn-gitlab-webservice-8181] <redacted>:8181 22 0.104 500 b819d3368bba94ad4f1f0650d59b568e 
[ldn-gitlab-webservice-69f47898cc-2wzsb webservice] {"time":"2022-03-24T18:13:48.265Z","severity":"INFO","duration_s":0.08831,"db_duration_s":0.00622,"view_duration_s":0.08209,"status":200,"method":"POST","path":"/api/v4/jobs/773702/artifacts/authorize","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"}],"host":"<redacted>.co.uk","remote_ip":"<redacted>","ua":"gitlab-runner 13.1.1 (13-1-stable; go1.13.8; linux/amd64)","route":"/api/:version/jobs/:id/artifacts/authorize","redis_calls":1,"redis_duration_s":0.000798,"redis_read_bytes":2,"redis_write_bytes":113,"redis_cache_calls":1,"redis_cache_duration_s":0.000798,"redis_cache_read_bytes":2,"redis_cache_write_bytes":113,"correlation_id":"b819d3368bba94ad4f1f0650d59b568e","meta.user":"xxx","meta.project":"<redacted>","meta.root_namespace":"<redacted>","meta.caller_id":"/api/:version/jobs/:id/artifacts/authorize"} 
[ldn-gitlab-webservice-69f47898cc-2wzsb gitlab-workhorse] time="2022-03-24T18:13:48Z" level=error msg=error correlation_id=sZPEEiD1dn3 error="handleFileUploads: extract files from multipart: persisting multipart file: create temporary buffer file: open /tmp/gitlab/part-buffer402913395: no such file or directory" method=POST uri="/api/v4/jobs/773702/artifacts?artifact_format=zip&artifact_type=archive"

Versions of components

Helm Chart version: 4.1.12
GitLab 13.1.11-ee (db98e78d9d7)
GitLab Shell 13.3.0
GitLab Workhorse v8.35.0
GitLab API  v4
Ruby   2.6.6p146
Rails  6.0.3.1
PostgreSQL 11.11

Not sure what to look for now?

Edited by dev ops