Runner won't upload artifacts to Azure object storage
Summary
I have just enabled Azure object storage for our GitLab instance using the consolidated configuration. Config snippet attached. When running a simple CI/CD pipeline, I get the following error:
WARNING: Uploading artifacts to coordinator... failed id=55 responseStatus=500 Internal Server Error status=500 Internal Server Error token=2P2x68Wq
WARNING: Retrying...
WARNING: Uploading artifacts to coordinator... failed id=55 responseStatus=500 Internal Server Error status=500 Internal Server Error token=2P2x68Wq
WARNING: Retrying...
WARNING: Uploading artifacts to coordinator... failed id=55 responseStatus=500 Internal Server Error status=500 Internal Server Error token=2P2x68Wq
FATAL: invalid argument
ERROR: Job failed: exit code 1
This pipeline worked before I enabled Azure object storage, and logs below definitely point to Azure storage being involved in the problem. The pipeline seems to fail after receiving 400 An HTTP header that's mandatory for this request is not specified
from a PUT
request to Azure object storage.
Steps to reproduce
I have a very simple repo that publishes to GitLab Pages. When running the pipeline, it fails as above.
.gitlab-ci.yml
pages:
stage: deploy
script:
- mkdir .public
- cp -r * .public
- mv .public public
artifacts:
paths:
- public
only:
- master
Actual behavior
Job fails with 500 Internal Server Error as above
Expected behavior
Job completes successfully and updates the Pages page
Relevant logs and/or screenshots
job log
Running with gitlab-runner 11.8.0 (4745a6f3)
on azae-gitlabce-01 7ead9ceb
Using Docker executor with image alpine:latest ...
Pulling docker image alpine:latest ...
Using docker image sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e for alpine:latest ...
Running on runner-7ead9ceb-project-69-concurrent-0 via azae-gitlabce-01...
00:03
Fetching changes...
00:02
Removing public/
HEAD is now at f39ee9f Update index.html
Checking out f39ee9f7 as master...
Skipping Git submodules setup
$ mkdir .public
00:02
$ cp -r * .public
$ mv .public public
Uploading artifacts...
00:07
public: found 3 matching files
WARNING: Uploading artifacts to coordinator... failed id=56 responseStatus=500 Internal Server Error status=500 Internal Server Error token=6ZeihmpX
WARNING: Retrying...
WARNING: Uploading artifacts to coordinator... failed id=56 responseStatus=500 Internal Server Error status=500 Internal Server Error token=6ZeihmpX
WARNING: Retrying...
WARNING: Uploading artifacts to coordinator... failed id=56 responseStatus=500 Internal Server Error status=500 Internal Server Error token=6ZeihmpX
FATAL: invalid argument
ERROR: Job failed: exit code 1
gitlab-workhorse/current
==> /var/log/gitlab/gitlab-workhorse/current <==
{"copied_bytes":1391,"correlation_id":"XgaEDcJI8K2","is_local":false,"is_multipart":false,"is_remote":true,"level":"info","msg":"saved file","remote_id":"1599451338-61797-0003-1559-be13879aeb90b0f273bc07dbe16e69ca","temp_file_prefix":"artifacts.zip","time":"2020-09-07T16:02:18+12:00","use_s3_client":false}
{"correlation_id":"XgaEDcJI8K2","error":"handleFileUploads: extract files from multipart: persisting multipart file: PUT request https://accountnamechanged.blob.core.windows.net/artifacts/tmp/uploads/1599451338-61797-4358-1559-be13879aeb95886273bc07dbe16e69ca?sp=c\u0026sv=2016-05-31\u0026sr=b\u0026se=2020-09-07T08%3A17%3A18Z\u0026spr=https\u0026sig=DsjHFxse%2Br3NrLuh1NCw5A1FlXXtFicpL8tk9fw%3D returned: 400 An HTTP header that's mandatory for this request is not specified.","level":"error","method":"POST","msg":"error","time":"2020-09-07T16:02:18+12:00","uri":"/api/v4/jobs/55/artifacts?artifact_format=zip\u0026artifact_type=archive"}
{"correlation_id":"XgaEDcJI8K2","duration_ms":61,"host":"gitlab.mycompany.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2020-09-07T16:02:18+12:00","uri":"/api/v4/jobs/55/artifacts?artifact_format=zip\u0026artifact_type=archive","user_agent":"gitlab-runner 11.8.0 (11-8-stable; go1.8.7; linux/amd64)","written_bytes":22}
grep -r 'status":500' /var/log/gitlab
gitlab-workhorse/current:{"correlation_id":"iLTzvde5383","duration_ms":1568,"host":"gitlab.mycompany.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2020-09-08T09:36:37+12:00","uri":"/api/v4/jobs/56/artifacts?artifact_format=zip\u0026artifact_type=archive","user_agent":"gitlab-runner 11.8.0 (11-8-stable; go1.8.7; linux/amd64)","written_bytes":22}
gitlab-workhorse/current:{"correlation_id":"MSB3vM1WYn9","duration_ms":49,"host":"gitlab.mycompany.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2020-09-08T09:36:38+12:00","uri":"/api/v4/jobs/56/artifacts?artifact_format=zip\u0026artifact_type=archive","user_agent":"gitlab-runner 11.8.0 (11-8-stable; go1.8.7; linux/amd64)","written_bytes":22}
gitlab-workhorse/current:{"correlation_id":"Sk8T5YLrjR6","duration_ms":408,"host":"gitlab.mycompany.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2020-09-08T09:36:40+12:00","uri":"/api/v4/jobs/56/artifacts?artifact_format=zip\u0026artifact_type=archive","user_agent":"gitlab-runner 11.8.0 (11-8-stable; go1.8.7; linux/amd64)","written_bytes":22}
sudo grep -r 'iLTzvde5383' /var/log/gitlab
/var/log/gitlab/gitlab-workhorse/current:{"copied_bytes":1391,"correlation_id":"iLTzvde5383","is_local":false,"is_multipart":false,"is_remote":true,"level":"info","msg":"saved file","remote_id":"1599514596-93903-0001-2519-896148178cda5591a9577ae2af0f0ed7","temp_file_prefix":"artifacts.zip","time":"2020-09-08T09:36:37+12:00","use_s3_client":false}
/var/log/gitlab/gitlab-workhorse/current:{"correlation_id":"iLTzvde5383","error":"handleFileUploads: extract files from multipart: persisting multipart file: PUT request https://accountnamechanged.blob.core.windows.net/artifacts/tmp/uploads/1599514596-93903-0001-2519-896148178cda5591a9577ae2af0f0ed7?sp=c\u0026sv=2016-05-31\u0026sr=b\u0026se=2020-09-08T01%3A51%3A37Z\u0026spr=https\u0026sig=q9DE%2FMQpdy%2BnfTgeby7yQ6FC0IWiCY50oUz3YC2m0ow%3D returned: 400 An HTTP header that's mandatory for this request is not specified.","level":"error","method":"POST","msg":"error","time":"2020-09-08T09:36:37+12:00","uri":"/api/v4/jobs/56/artifacts?artifact_format=zip\u0026artifact_type=archive"}
/var/log/gitlab/gitlab-workhorse/current:{"correlation_id":"iLTzvde5383","duration_ms":1568,"host":"gitlab.mycompany.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2020-09-08T09:36:37+12:00","uri":"/api/v4/jobs/56/artifacts?artifact_format=zip\u0026artifact_type=archive","user_agent":"gitlab-runner 11.8.0 (11-8-stable; go1.8.7; linux/amd64)","written_bytes":22}
Environment description
Running self-hosted GitLab EE 13.3.5
object storage config
################
# Object Store #
################
# Consolidated object storage configuration
gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['proxy_download'] = false
gitlab_rails['object_store']['connection'] = {
'provider' => 'AzureRM',
'azure_storage_account_name' => 'accountnamechanged',
'azure_storage_access_key' => 'keyremoved'
}
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'artifacts'
gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'external-diffs'
gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'lfs-objects'
gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'uploads'
gitlab_rails['object_store']['objects']['packages']['bucket'] = 'packages'
gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'dependency-proxy'
gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'terraform-state'