After upgrade to 13.9.0 (helm chart v 4.9.0) artifacts upload to S3 stopped working
Summary
After upgrade to 13.9.0
artifacts upload to S3 stopped working.
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed id=725457 responseStatus=500 Internal Server Error status=500 token=JCu_ES63
FATAL: invalid argument
Steps to reproduce
Upgrade gitlab
helm chart from 4.8.4
to 4.9.0
. Used to work on 4.8.4
.
Configuration used
appConfig:
artifacts:
bucket: <s3_bucket>
connection:
secret: <aws_secret_object_storage>
key: connection
Current behavior
Job logs:
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed id=725457 responseStatus=500 Internal Server Error status=500 token=JCu_ES63
FATAL: invalid argument
workhorse logs:
$ kubectl logs -f --tail=10 -c gitlab-workhorse -l release=gitlab-server,app=webservice | grep error
{"error":"RequestError: send request failed\ncaused by: Put \"https://<s3_bucket>.<s3_bucket>.s3.amazonaws.com/tmp/uploads/1614023451-27-0041-4656-a140af75d46e108de62e2b205c4b3f4f\": x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not <s3_bucket>.<s3_bucket>.s3.amazonaws.com","level":"error","msg":"error uploading S3 session","time":"2021-02-22T19:50:53Z"}
{"correlation_id":"01EZ5MPRASRFTRDRZDXH6SEBFW","error":"handleFileUploads: extract files from multipart: persisting multipart file: Put \"https://<s3_bucket>.<s3_bucket>.s3.amazonaws.com/tmp/uploads/1614023451-27-0041-4656-a140af75d46e108de62e2b205c4b3f4f\": x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not <s3_bucket>.<s3_bucket>.s3.amazonaws.com","level":"error","method":"POST","msg":"Internal server error","time":"2021-02-22T19:50:53Z","uri":"/api/v4/jobs/725513/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=1+week"}
Expected behavior
Artifacts can be uploaded to S3.
Versions
- Chart: gitlab-4.9.0
- Platform:
- Cloud: AWS EKS
- Kubernetes:
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-12T01:09:16Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
- Helm:
version.BuildInfo{Version:"v3.5.0", GitCommit:"32c22239423b3b4ba6706d450bd044baffdcf9e6", GitTreeState:"dirty", GoVersion:"go1.15.6"}
Relevant logs
Job logs:
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed id=725457 responseStatus=500 Internal Server Error status=500 token=JCu_ES63
FATAL: invalid argument
workhorse logs:
$ kubectl logs -f --tail=10 -c gitlab-workhorse -l release=gitlab-server,app=webservice | grep error
{"error":"RequestError: send request failed\ncaused by: Put \"https://<s3_bucket>.<s3_bucket>.s3.amazonaws.com/tmp/uploads/1614023451-27-0041-4656-a140af75d46e108de62e2b205c4b3f4f\": x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not <s3_bucket>.<s3_bucket>.s3.amazonaws.com","level":"error","msg":"error uploading S3 session","time":"2021-02-22T19:50:53Z"}
{"correlation_id":"01EZ5MPRASRFTRDRZDXH6SEBFW","error":"handleFileUploads: extract files from multipart: persisting multipart file: Put \"https://<s3_bucket>.<s3_bucket>.s3.amazonaws.com/tmp/uploads/1614023451-27-0041-4656-a140af75d46e108de62e2b205c4b3f4f\": x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not <s3_bucket>.<s3_bucket>.s3.amazonaws.com","level":"error","method":"POST","msg":"Internal server error","time":"2021-02-22T19:50:53Z","uri":"/api/v4/jobs/725513/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=1+week"}
Workaround
Add endpoint
to the connection block:
# object-storage.yaml
provider: AWS
use_iam_profile: true
region: us-east-1
# workaround for https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2589
endpoint: https://s3.amazonaws.com
Edited by Stan Hu