Artifact upload broken after update to 2.4.1
Summary
After upgrading from the latest 2.3 chart to 2.4.0, artifact upload was broken. This was reported and fixed as #1647 (closed) in 2.4.1. With 2.4.1, likely 2.4.0 but masked by the aforementioned issue, upload is broken, though not with a 500 but with a 400.
Steps to reproduce
Making a deployment with artifacts to upload to object storage. I am using DigitalOcean Spaces (S3 compatible), which may or may not have impact.
Configuration used
Same as for the 2.3 chart:
artifacts:
bucket: gitlab-artifacts
connection:
secret: gitlab-object
key: connection
With a secret
provider: AWS
region: fra1
aws_access_key_id: ACCESS KEY
aws_secret_access_key: SECRET
aws_signature_version: 4
endpoint: "https://fra1.digitaloceanspaces.com"
The configuration was working with 2.3.
Current behavior
When building, I get error during artifact upload:
Uploading artifacts...
public: found 32 matching files
WARNING: Uploading artifacts to coordinator... failed id=20629 responseStatus=400 Bad Request status=400 Bad Request token=3mUSwQmy
WARNING: Retrying... error=invalid argument
WARNING: Uploading artifacts to coordinator... failed id=20629 responseStatus=400 Bad Request status=400 Bad Request token=3mUSwQmy
WARNING: Retrying... error=invalid argument
WARNING: Uploading artifacts to coordinator... failed id=20629 responseStatus=400 Bad Request status=400 Bad Request token=3mUSwQmy
FATAL: invalid argument
ERROR: Job failed: command terminated with exit code 1
I tried and I can still see, browse and download artifacts produced before the upgrade.
Expected behavior
Uploading working.
Versions
- Chart: 2.4.1
- Platform:
- Cloud: DigitalOcean managed Kubernetes
- Kubernetes: (
kubectl version
)- Client Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.9-dispatcher", GitCommit:"e3f5193e8f1091a162af7e17a781e6a3129bcfd0", GitTreeState:"clean", BuildDate:"2019-03-28T18:15:47Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"darwin/amd64"}
- Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
- Helm: (
helm version
)- Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
- Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Relevant logs
I see very little in production.log:
Started POST "/api/v4/jobs/20627/artifacts?artifact_format=zip&artifact_type=archive" for 10.244.0.15 at 2019-10-24 09:37:18 +0000
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Parameters: {"file.sha512"=>"3bbd01164b5edb9a9d05960425af6668601f725b5b1a315f39a479785a84e3275fed62ae8965e9de53ac70e11b9aa6f3a1845bd8bea4b358afcbf39b6a8772ef", "file.etag"=>"a9858e2b37f3940294383a24fe61ee5c-1", "file.name"=>"artifacts.zip", "file.remote_url"=>"https://gitlab-artifacts.fra1.digitaloceanspaces.com/tmp/uploads/1571909837-51-0002-4555-cd277a0b0d456be8626a0f5eec2a9712?X-Amz-Expires=15300&X-Amz-Date=20191024T093717Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Z73PHGNXXNWQE4WKRKU5%2F20191024%2Ffra1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=a6b1b450de5f00e768c01e080387d452b4b26eb462b06f2077e4c6afbaa86f1a", "file.size"=>"49286", "file.sha256"=>"27b91717f1e7fffb1180fcde8553e842664dbf5a48ef3fbb69cc1059caf7cf4a", "file.remote_id"=>"1571909837-51-0002-4555-cd277a0b0d456be8626a0f5eec2a9712", "file.md5"=>"614a1f526f080be5fa6a3bfc0502fb6d", "file.sha1"=>"61e46d262fd4c6492358c48a6a2ede0190be3f2c", "metadata.name"=>"metadata.gz", "metadata.path"=>"/tmp/metadata.gz973158300", "metadata.size"=>"927", "metadata.sha1"=>"4ce59583de32b78af29af4dfb9d7f2473bd2cb91", "metadata.sha256"=>"b1ca5b632c846422d36766f57e5a4f635fe0eb4b37240e001a30d793d4937530", "metadata.sha512"=>"4e7f53feb24405472d9026f994c3a7ccf48d1c413e88aa9547d31b3a9e6f1b9b06145005b93f1127d4963f1d6c2cb50e3ad6a9bbbebf7fe88e3c66d6ef2345a6", "metadata.md5"=>"39e476d93784b6c2a27a4b61bf1ec547", "artifact_format"=>"zip", "artifact_type"=>"archive"}
Can't verify CSRF token authenticity.
This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection`
Unlike the logs may suggest, this does not result in an actual 422 response to the user
For API requests, the only effect is that `current_user` will be `nil` for the duration of the request
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
Processing by Projects::JobsController#trace as JSON
Parameters: {"state"=>"eyJvZmZzZXQiOjE1OTkzOSwibl9vcGVuX3RhZ3MiOjAsImZnX2NvbG9yIjpudWxsLCJiZ19jb2xvciI6bnVsbCwic3R5bGVfbWFzayI6MCwic2VjdGlvbnMiOlsiYnVpbGQtc2NyaXB0Il0sImxpbmVub19pbl9zZWN0aW9uIjoxNTQzfQ==", "namespace_id"=>"group", "project_id"=>"project", "id"=>"20627"}
Completed 200 OK in 78ms (Views: 1.3ms | ActiveRecord: 15.4ms | Elasticsearch: 0.0ms)
(I've mildly anonymized the bucket name).
In api_json.log I see something that looks related:
{"time":"2019-10-24T09:37:15.014Z","severity":"INFO","duration":13.1,"db":4.33,"view":8.77,"status":400,"method":"POST","path":"/api/v4/jobs/20627/artifacts","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"},{"key":"file.sha1","value":"61e46d262fd4c6492358c48a6a2ede0190be3f2c"},{"key":"file.sha256","value":"27b91717f1e7fffb1180fcde8553e842664dbf5a48ef3fbb69cc1059caf7cf4a"},{"key":"file.name","value":"artifacts.zip"},{"key":"file.remote_url","value":"https://gitlab-artifacts.fra1.digitaloceanspaces.com/tmp/uploads/1571909834-52-0001-4225-f6e62a6b4c7820519ae28719ae30e69a?X-Amz-Expires=15300&X-Amz-Date=20191024T093714Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Z73PHGNXXNWQE4WKRKU5%2F20191024%2Ffra1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=0e9074dbe7d4112c4592dad95371aa6a58cb2501fef3b74bc6010657cc1f11fe"},{"key":"file.remote_id","value":"1571909834-52-0001-4225-f6e62a6b4c7820519ae28719ae30e69a"},{"key":"file.size","value":"49286"},{"key":"file.sha512","value":"3bbd01164b5edb9a9d05960425af6668601f725b5b1a315f39a479785a84e3275fed62ae8965e9de53ac70e11b9aa6f3a1845bd8bea4b358afcbf39b6a8772ef"},{"key":"file.md5","value":"614a1f526f080be5fa6a3bfc0502fb6d"},{"key":"file.etag","value":"a9858e2b37f3940294383a24fe61ee5c-1"},{"key":"metadata.name","value":"metadata.gz"},{"key":"metadata.path","value":"/tmp/metadata.gz180232902"},{"key":"metadata.size","value":"927"},{"key":"metadata.sha512","value":"4e7f53feb24405472d9026f994c3a7ccf48d1c413e88aa9547d31b3a9e6f1b9b06145005b93f1127d4963f1d6c2cb50e3ad6a9bbbebf7fe88e3c66d6ef2345a6"},{"key":"metadata.md5","value":"39e476d93784b6c2a27a4b61bf1ec547"},{"key":"metadata.sha1","value":"4ce59583de32b78af29af4dfb9d7f2473bd2cb91"},{"key":"metadata.sha256","value":"b1ca5b632c846422d36766f57e5a4f635fe0eb4b37240e001a30d793d4937530"}],"host":"gitlab.example.com","remote_ip":"10.133.230.149, 10.244.0.15","ua":"gitlab-runner 12.4.0 (12-4-stable; go1.10.8; linux/amd64)","route":"/api/:version/jobs/:id/artifacts","queue_duration":31.45,"correlation_id":"j4WWxQBr893"}