Skip to content

Job succeed, but uploading artifacts fails with 500 error

Summary

hosted Gitlab CE 13.5 on one server, on another server gitlab-runner 13.5 with shell runner.

Some tests are running with docker -v, so after successful run I have files owned by root and with permissions 0600.

Gitlab runner is running by default under gitlab-runner user.

After successful running tests gitlab-runner cannot upload log results to coordinator.

Here is what I get in gitlab web UI:

Tests skipped: 0
Flaky tests skipped: 0
Tests ok: 16
Updating /var/log/index.html ... done
Updating /var/log/all_runs.html ... done
retval: '0'
Uploading artifacts for successful job
00:04
Uploading artifacts...
Runtime platform                                    arch=amd64 os=linux pid=182971 revision=738bbe5a version=13.3.1
test-logs/*: found 86 matching files and directories 
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=326076 responseStatus=500 Internal Server Error status=500 token=JaJAAzyk
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=326076 responseStatus=500 Internal Server Error status=500 token=JaJAAzyk
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=326076 responseStatus=500 Internal Server Error status=500 token=JaJAAzyk
FATAL: invalid argument                            
ERROR: Job failed: exit status 1

Here what I can find on gitlab server:

root@git2:/var/log/gitlab# grep -r 'status":500' .
...
./gitlab-workhorse/current:{"correlation_id":"4SUqPjBRQX3","duration_ms":24,"host":"my.gitlab.local","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"172.16.0.15:0","remote_ip":"172.16.0.15","status":500,"system":"http","time":"2020-09-05T15:32:00+03:00","uri":"/api/v4/jobs/326076/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=1+week","user_agent":"gitlab-runner 13.3.1 (13-3-stable; go1.13.8; linux/amd64)","written_bytes":22}

root@git2:/var/log/gitlab# grep -r 4SUqPjBRQX3 .
./gitlab-workhorse/current:{"correlation_id":"4SUqPjBRQX3","error":"handleFileUploads: extract files from multipart: persisting multipart file: unexpected EOF","level":"error","method":"POST","msg":"error","time":"2020-09-05T15:32:00+03:00","uri":"/api/v4/jobs/326076/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=1+week"}
./gitlab-workhorse/current:{"correlation_id":"4SUqPjBRQX3","duration_ms":24,"host":"my.gitlab.local","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"172.16.0.15:0","remote_ip":"172.16.0.15","status":500,"system":"http","time":"2020-09-05T15:32:00+03:00","uri":"/api/v4/jobs/326076/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=1+week","user_agent":"gitlab-runner 13.3.1 (13-3-stable; go1.13.8; linux/amd64)","written_bytes":22}

When I run gitlab-runner from root, everything works ok.

Steps to reproduce

Run tests in docker with docker -v and create file, owned by root with permission 0600


tests:
  stage: tests
  script:
    - mkdir -p test-logs
    - docker run -v `pwd`/test-logs:/var/log /run-tests.sh
  artifacts:
    reports:
      junit: test-logs/ct_run*/junit_report.xml
    when: always
    name: "tests"
    expire_in: 1 week
    paths:
      - test-logs/*

Actual behavior

gitlab-runner starts uploading files, then it silently breaks upload and terminate connection. gitlab shows 500 error

Expected behavior

For example, writing into logs that some file cannot be uploaded.

Relevant logs and/or screenshots

See upper

Environment description

See upper

Used GitLab Runner version

# gitlab-runner --version
Version:      13.3.1
Git revision: 738bbe5a
Git branch:   13-3-stable
GO version:   go1.13.8
Built:        2020-08-25T12:29:06+0000
OS/Arch:      linux/amd64

Possible fixes

More verbose checking file permissions during uploading directory based artifacts.

Proposal for solution implementation

  • {placeholder}
Edited by Darren Eastman