Job succeed, but uploading artifacts fails with 500 error
Summary
hosted Gitlab CE 13.5 on one server, on another server gitlab-runner 13.5 with shell runner.
Some tests are running with docker -v, so after successful run I have files owned by root and with permissions 0600.
Gitlab runner is running by default under gitlab-runner user.
After successful running tests gitlab-runner cannot upload log results to coordinator.
Here is what I get in gitlab web UI:
Tests skipped: 0
Flaky tests skipped: 0
Tests ok: 16
Updating /var/log/index.html ... done
Updating /var/log/all_runs.html ... done
retval: '0'
Uploading artifacts for successful job
00:04
Uploading artifacts...
Runtime platform arch=amd64 os=linux pid=182971 revision=738bbe5a version=13.3.1
test-logs/*: found 86 matching files and directories
WARNING: Uploading artifacts as "archive" to coordinator... failed id=326076 responseStatus=500 Internal Server Error status=500 token=JaJAAzyk
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed id=326076 responseStatus=500 Internal Server Error status=500 token=JaJAAzyk
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed id=326076 responseStatus=500 Internal Server Error status=500 token=JaJAAzyk
FATAL: invalid argument
ERROR: Job failed: exit status 1
Here what I can find on gitlab server:
root@git2:/var/log/gitlab# grep -r 'status":500' .
...
./gitlab-workhorse/current:{"correlation_id":"4SUqPjBRQX3","duration_ms":24,"host":"my.gitlab.local","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"172.16.0.15:0","remote_ip":"172.16.0.15","status":500,"system":"http","time":"2020-09-05T15:32:00+03:00","uri":"/api/v4/jobs/326076/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=1+week","user_agent":"gitlab-runner 13.3.1 (13-3-stable; go1.13.8; linux/amd64)","written_bytes":22}
root@git2:/var/log/gitlab# grep -r 4SUqPjBRQX3 .
./gitlab-workhorse/current:{"correlation_id":"4SUqPjBRQX3","error":"handleFileUploads: extract files from multipart: persisting multipart file: unexpected EOF","level":"error","method":"POST","msg":"error","time":"2020-09-05T15:32:00+03:00","uri":"/api/v4/jobs/326076/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=1+week"}
./gitlab-workhorse/current:{"correlation_id":"4SUqPjBRQX3","duration_ms":24,"host":"my.gitlab.local","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"172.16.0.15:0","remote_ip":"172.16.0.15","status":500,"system":"http","time":"2020-09-05T15:32:00+03:00","uri":"/api/v4/jobs/326076/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=1+week","user_agent":"gitlab-runner 13.3.1 (13-3-stable; go1.13.8; linux/amd64)","written_bytes":22}
When I run gitlab-runner from root, everything works ok.
Steps to reproduce
Run tests in docker with docker -v and create file, owned by root with permission 0600
tests:
stage: tests
script:
- mkdir -p test-logs
- docker run -v `pwd`/test-logs:/var/log /run-tests.sh
artifacts:
reports:
junit: test-logs/ct_run*/junit_report.xml
when: always
name: "tests"
expire_in: 1 week
paths:
- test-logs/*
Actual behavior
gitlab-runner starts uploading files, then it silently breaks upload and terminate connection. gitlab shows 500 error
Expected behavior
For example, writing into logs that some file cannot be uploaded.
Relevant logs and/or screenshots
See upper
Environment description
See upper
Used GitLab Runner version
# gitlab-runner --version
Version: 13.3.1
Git revision: 738bbe5a
Git branch: 13-3-stable
GO version: go1.13.8
Built: 2020-08-25T12:29:06+0000
OS/Arch: linux/amd64
Possible fixes
More verbose checking file permissions during uploading directory based artifacts.
Proposal for solution implementation
- {placeholder}