Error code 400 when uploading a large artifact (AWS S3)
Summary
A customer is encountering this error on GitLab CI when trying to upload a large artifact:
WARNING: Uploading artifacts to coordinator... failed id=81 responseStatus=400 Bad Request status=400 Bad Request token=cbuTvhg1
WARNING: Retrying... error=invalid argumentI managed to reproduce in GitLab 12.9.0 and using AWS S3 as the storage for the artifact. When using Google Cloud Storage, it is working properly.
I have the following settings:
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_direct_upload'] = true
gitlab_rails['artifacts_object_store_background_upload'] = false
gitlab_rails['artifacts_object_store_remote_directory'] = "jdasmarinas-test-bucket"
gitlab_rails['artifacts_object_store_connection'] = {
'provider' => 'AWS',
'aws_access_key_id' => '****',
'aws_secret_access_key' => '****',
'region' => 'ap-southeast-1',
}Steps to reproduce
- Configure artifacts to use AWS S3. Enable
direct_upload. - Try to upload a large artifact. I used this
.gitlab-ci.ymlfor testing:
stages:
- build
build:
stage: build
script:
- echo "Hello world!"
- dd if=/dev/urandom of=./testfile.block bs=1024K count=6024
artifacts:
paths:
- ./testfile.block
expire_in: 1 weekWhat is the current bug behavior?
This shows at the end of the ci logs:
WARNING: Uploading artifacts to coordinator... failed id=81 responseStatus=400 Bad Request status=400 Bad Request token=cbuTvhg1
WARNING: Retrying... error=invalid argumentThen this is the relevant log for api_json.log:
{
"time": "2020-03-26T10:53:56.889Z",
"severity": "INFO",
"duration": 357.55,
"db": 23.36,
"view": 334.19,
"status": 400,
"method": "POST",
"path": "/api/v4/jobs/81/artifacts",
"params": [
{
"key": "artifact_format",
"value": "zip"
},
{
"key": "artifact_type",
"value": "archive"
},
{
"key": "expire_in",
"value": "1 week"
},
{
"key": "file.name",
"value": "artifacts.zip"
},
{
"key": "file.sha512",
"value": "e68e74eefdff7a49cbbc314c430c3a0fc1123c78945e3b75823711bb2880a73ec80c0e3e3d537d3df8a03518c98fa8502d0acff95c64b581481a90474c345d75"
},
{
"key": "file.md5",
"value": "b605a80771aee1130c3d240a13308161"
},
{
"key": "file.etag",
"value": "3b25517a29c6a8149e49aa0f27fb9a22-87"
},
{
"key": "file.sha1",
"value": "767f3f9ef16ea7d2794a8523b76203f72a60ab57"
},
{
"key": "file.remote_url",
"value": "https://jdasmarinas-test-bucket.s3-ap-southeast-1.amazonaws.com/tmp/uploads/1585219814-21165-0001-5271-22885406f09ced5f3b177788413ef375?X-Amz-Expires=15300&X-Amz-Date=20200326T105015Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<SNIPPED>%2F20200326%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=35dc5caf01bb18db552c4e222b7d73eaee966bb180ff2ac7e5ad9af4fa1891fd"
},
{
"key": "file.remote_id",
"value": "1585219814-21165-0001-5271-22885406f09ced5f3b177788413ef375"
},
{
"key": "file.size",
"value": "6318549824"
},
{
"key": "file.sha256",
"value": "6e150b15fd17dbe896960567b021d92083083c1edaf53abe277d054cde7777f7"
},
{
"key": "metadata.path",
"value": "/tmp/metadata.gz910136994"
},
{
"key": "metadata.size",
"value": "169"
},
{
"key": "metadata.sha512",
"value": "f4ded8e012f7d45ed5c8c9b00068038fafaf72d4d6b774ec26c539ba0c0db9a2f0ec1c75e75a4aabd359d548435f39df16642ba5d47b7004e44f0dd580bf8c79"
},
{
"key": "metadata.md5",
"value": "c8e343405007f62f5a96b14c0bf7ab85"
},
{
"key": "metadata.sha1",
"value": "b43777a36030b601812ef9a78ea7e090403cc003"
},
{
"key": "metadata.sha256",
"value": "2f8078e28af04d9e362033e6680049687f4b13fdb57e5438fe696af95b58f98a"
},
{
"key": "metadata.name",
"value": "metadata.gz"
}
],
"host": "jdasmarinas-geo-01.do.gitlap.com",
"remote_ip": "128.199.225.37, 128.199.225.37",
"ua": "gitlab-runner 12.8.0 (12-8-stable; go1.13.7; linux/amd64)",
"route": "/api/:version/jobs/:id/artifacts",
"queue_duration": 26.38,
"correlation_id": "OF5y9mcEOq5"
}Job logs (this is based on my test):
Running with gitlab-runner 12.8.0 (1b659122)
on docker 9yxL-HDW
Using Docker executor with image alpine:3 ...
00:04
Pulling docker image alpine:3 ...
Using docker image sha256:a187dde48cd289ac374ad8539930628314bc581a481cdb41409c9289419ddb72 for alpine:3 ...
Running on runner-9yxL-HDW-project-20-concurrent-0 via jdasmarinas-runner-01...
00:01
Fetching changes with git depth set to 50...
00:02
Reinitialized existing Git repository in /builds/root/large-artifact/.git/
From https://jdasmarinas-geo-01.do.gitlap.com/root/large-artifact
* [new ref] refs/pipelines/33 -> refs/pipelines/33
Checking out fbe71192 as master...
Removing testfile.block
Skipping Git submodules setup
$ echo "Hello world!"
00:55
Hello world!
$ dd if=/dev/urandom of=./testfile.block bs=1024K count=6024
6024+0 records in
6024+0 records out
Uploading artifacts...
30:47
./testfile.block: found 1 matching files
WARNING: Uploading artifacts to coordinator... failed id=81 responseStatus=400 Bad Request status=400 Bad Request token=cbuTvhg1
WARNING: Retrying... error=invalid argument
WARNING: Uploading artifacts to coordinator... failed id=81 responseStatus=400 Bad Request status=400 Bad Request token=cbuTvhg1
WARNING: Retrying... error=invalid argument
WARNING: Uploading artifacts to coordinator... failed id=81 responseStatus=400 Bad Request status=400 Bad Request token=cbuTvhg1
FATAL: invalid argument
ERROR: Job failed: exit code 1What is the expected correct behavior?
Large artifacts should be uploaded properly
Output of checks
Results of GitLab environment info
# gitlab-rake gitlab:env:info
System information
System: Debian 9.12
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.6.5p114
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.7
Git Version: 2.24.1
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 12.9.0-ee
Revision: 073a4ba8016
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 9.6.11
URL: https://gitlab.sp-insticloud.de
HTTP Clone URL: https://*****/some-group/some-project.git
SSH Clone URL: git@****:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 12.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/gitResults of GitLab application Check
# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 12.0.0 ? ... OK (12.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/2 ... yes
3/5 ... yes
3/6 ... yes
3/7 ... yes
3/8 ... yes
3/9 ... yes
3/10 ... yes
3/11 ... yes
3/12 ... yes
3/13 ... yes
3/14 ... yes
3/15 ... yes
3/16 ... yes
3/17 ... yes
3/18 ... yes
3/19 ... yes
3/20 ... yes
3/21 ... yes
3/22 ... yes
3/23 ... yes
3/24 ... yes
3/25 ... yes
3/26 ... yes
3/27 ... yes
3/28 ... yes
3/29 ... yes
3/30 ... yes
3/31 ... yes
3/32 ... yes
3/33 ... yes
3/34 ... yes
3/35 ... yes
3/36 ... yes
3/37 ... yes
3/38 ... yes
3/39 ... yes
3/40 ... yes
3/41 ... yes
3/42 ... yes
3/43 ... yes
3/45 ... yes
3/46 ... yes
3/48 ... yes
3/49 ... yes
22/50 ... yes
23/51 ... yes
3/52 ... yes
23/53 ... yes
3/54 ... yes
3/55 ... yes
3/56 ... yes
3/57 ... yes
3/58 ... yes
3/59 ... yes
3/60 ... yes
3/61 ... yes
3/62 ... yes
3/63 ... yes
3/64 ... yes
3/65 ... yes
3/68 ... yes
3/69 ... yes
34/70 ... yes
3/71 ... yes
3/73 ... yes
35/74 ... yes
35/75 ... yes
35/76 ... yes
3/77 ... yes
3/78 ... yes
3/79 ... yes
37/80 ... yes
37/81 ... yes
37/82 ... yes
37/83 ... yes
37/85 ... yes
35/86 ... yes
37/87 ... yes
37/88 ... yes
37/89 ... yes
37/90 ... yes
37/91 ... yes
37/92 ... yes
3/93 ... yes
35/94 ... yes
37/96 ... yes
3/97 ... yes
39/98 ... yes
3/99 ... yes
3/100 ... yes
37/101 ... yes
37/102 ... yes
37/103 ... yes
37/104 ... yes
37/105 ... yes
39/106 ... yes
37/107 ... yes
39/108 ... yes
39/109 ... yes
39/110 ... yes
43/111 ... yes
39/112 ... yes
43/113 ... yes
39/114 ... yes
37/115 ... yes
43/116 ... yes
37/117 ... yes
43/118 ... yes
37/119 ... yes
35/120 ... yes
37/121 ... yes
35/122 ... yes
35/123 ... yes
16/124 ... yes
37/125 ... yes
37/126 ... yes
35/127 ... yes
54/128 ... yes
54/129 ... yes
39/130 ... yes
55/131 ... yes
37/132 ... yes
37/133 ... yes
37/134 ... yes
55/135 ... yes
35/136 ... yes
37/137 ... yes
59/138 ... yes
37/139 ... yes
35/140 ... yes
35/141 ... yes
2/142 ... yes
35/143 ... yes
43/144 ... yes
43/145 ... yes
35/147 ... yes
39/148 ... yes
56/149 ... yes
35/150 ... yes
3/151 ... yes
37/152 ... yes
43/153 ... yes
39/154 ... yes
35/155 ... yes
35/156 ... yes
3/157 ... yes
35/158 ... yes
55/159 ... yes
3/160 ... yes
35/161 ... yes
35/162 ... yes
62/163 ... yes
3/164 ... yes
3/165 ... yes
3/167 ... yes
3/168 ... yes
35/169 ... yes
3/170 ... yes
35/171 ... yes
35/172 ... yes
59/173 ... yes
55/176 ... yes
16/177 ... yes
3/178 ... yes
62/179 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.24.1)
Git user has default SSH configuration? ... yes
Active users: ... 35
Is authorized keys file accessible? ... yes
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... FinishedCustomer info
ZD Link (internal): https://gitlab.zendesk.com/agent/tickets/150617
Customer's version_manifest.txt:
gitlab-ee 12.9.0
Component Installed Version Version GUID
----------------------------------------------------------------------------------------------------------------------------------------------------
acme-client 2.0.6
alertmanager v0.20.0 git:0c0c6bdb0133e399c1a9fe4c1a170ce49ec67b58
awesome_print 1.8.0
bundler 1.17.3
bzip2 1.0.6 md5:00b516f4704d4a7cb50a1d97e6e8e15b
cacerts 2019.01.23 md5:fccbe6cec7a76e3351ad32e305184787
chef-acme v4.1.1 git:d39387e6ad0ee7dcbc1e08a438a1c616479e1300
chef-gem 14.14.29
chef-zero 14.0.13
compat_resource v12.19.1 git:e36200f6b804915b68a4ce74c8b7a293c041d9fe
config_guess c9092d05347c925a26f6887980e185206e13f9d6 git:5256817ace8493502ec88501a19e4051c2e220b0
consul v1.6.1 git:a103c62f33e9adba98302db140520a73e2b7d3e1
curl curl-7_59_0 git:4a90c5b82e21821496ef1afb5210a13f2564fcac
docker-distribution-pruner v0.1.0 git:800e59b025e7376ae0dec92454b458bb504b9919
exiftool 11.70 git:1bd3a2086f47b950212a094e70c7b2dd7696c1ac
git 2.24.1 sha256:ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02
gitaly v12.9.0-rc42 git:b4aaf619233129a8627cc2d2c96a310c972127e4
gitlab-config-template 12.9.0
gitlab-cookbooks 12.9.0
gitlab-ctl 12.9.0
gitlab-ctl-ee 12.9.0
gitlab-elasticsearch-indexer v2.1.0 git:20112ba0456c414f80dca43d7991074856c5b748
gitlab-exporter 6.1.0
gitlab-geo-psql bc339ea21bdbb329263ad93788aaa428
gitlab-healthcheck 90e6447aeead4a29ac9fffc15945ce6b
gitlab-pages v1.17.0 git:68ba5c768ba8374be11cff13c0815d4eb3bd4571
gitlab-pg-ctl 71f87e0dc829f288dfbec8991cc43675
gitlab-psql 7e11364159031db0eb55867ad3d5713b
gitlab-rails v12.9.0-ee git:073a4ba8016203c9e48595afd3ddea430d9420a3
gitlab-scripts 12.9.0
gitlab-selinux 12.9.0
gitlab-shell v12.0.0 git:6e4495c01509b20144b3b33d92a06f0a7a93e60a
gitlab-workhorse v8.25.0 git:3f55999aba13300402e0c7bfdef15da0ec5a8702
gnupg 2.2.10 sha256:799dd37a86a1448732e339bd20440f4f5ee6e69755f6fd7a73ee8af30840c915
go-crond 0.6.1 git:8b21a6742dee4d2cfad54079f9b579a4efcd0ad6
gpgme 1.10.0 sha256:1a8fed1197c3b99c35f403066bb344a26224d292afc048cfdfc4ccd5690a0693
grafana 6.3.5 sha256:0104bfe14444cea2fa3f021b9a75fc78f66434f2ca8f3d0bdd422d108ce682e7
grafana-dashboards v1.3.0 git:6404703224372df8994e27fbcffd400fb002b00a
graphicsmagick 1.3.34 sha256:4717f7a32d964c515d83706fd52d34e089c2ffa35f8fbf43c923ce19343cf2f4
jemalloc 5.2.1 git:2abb02ecd74e7e65d3992a542ffb43abe91a8a7f
krb5 krb5-1.17 git:fcfb0e47c995a7e9f956c3716be3175f44ad26e0
libassuan 2.5.1 sha256:47f96c37b4f2aac289f0bc1bacfa8bd8b4b209a488d3d15e2229cb6cc9b26449
libedit 20120601-3.0 md5:e50f6a7afb4de00c81650f7b1a0f5aea
libevent release-2.1.8-stable git:6f970267b6ba68c9dd9090d789c928529745dc68
libffi 3.2.1 md5:83b89587607e3eb65c70d361f13bab43
libgcrypt 1.8.3 sha256:66ec90be036747602f2b48f98312361a9180c97c68a690a5f376fa0f67d0af7c
libgpg-error 1.32 sha256:c345c5e73cc2332f8d50db84a2280abfb1d8f6d4f1858b9daa30404db44540ca
libiconv 1.15 sha256:ccf536620a45458d26ba83887a983b96827001e92a13847b45e4925cc8913178
libicu release-57-1 git:b772241b52fcec07b1909d44d156fbcd0cbfb20c
libksba 1.3.5 sha256:41444fd7a6ff73a79ad9728f985e71c9ba8cd3e5e53358e70d5f066d35c1a340
liblzma 5.2.4 sha256:b512f3b726d3b37b6dc4c8570e137b9311e7552e8ccbab4d39d47ce5f4177145
libossp-uuid 1.6.2 md5:5db0d43a9022a6ebbbc25337ae28942f
libpng v1.6.37 git:301f7a14295a3bdfaf406dbb5004d0784dc137ea
libre2 2016-02-01 git:5bd613749fd530b576b890283bfb6bc6ea6246cb
libtool 2.4.6 sha256:e3bd4d5d3d025a36c21dd6af7ea818a2afcd4dfc1ea5a17b39d7854bcd0c06e3
libxml2 2.9.9 md5:c04a5a0a042eaa157e8e8c9eabe76bd6
libxslt 1.1.32 md5:1fc72f98e98bf4443f1651165f3aa146
libyaml 0.1.7 sha256:8088e457264a98ba451a90b8661fcb4f9d6f478f7265d48322a196cec2480729
logrotate 3.15.1 git:556599800eb8e2f4d6d53c8ada413fda95f63bbb
mattermost 5.20.1 md5:8fb20bda9ac0fae60f4a09bec019314b
mixlib-log 3.0.1
ncurses 5.9 md5:8cb9c412e5f2d96bc6f459aa8c6282a1
nginx 1.16.1 sha256:f11c2a6dd1d3515736f0324857957db2de98be862461b5a542a3ac6188dbe32b
nginx-module-vts v0.1.18 git:46d85558e344dfe2b078ce757fd36c69a1ec2dd3
node-exporter v0.18.1 git:48bb6f670c6b8e1755312d19ab03686f8d555916
npth 1.6 sha256:1393abd9adcf0762d34798dc34fdcf4d0d22a8410721e76f1e3afcd1daa4e2d1
ohai 14.14.0
omnibus-ctl 1b96ac486636cac987e5b464810bb3ff673a93fe git:e160156deaa0afc037748b10d1c9ef0a5be97dc1
openssl OpenSSL_1_1_1d git:9a1c170d6309bb814ba8d720503069337f628b32
package-scripts 12.9.0+ee.0
pcre 8.42 md5:fc18afa0f14a25475cf097ee102a3e4f
pcre2 10.31 md5:10e1a95a0f0d80759f1b3d70e92c2cda
pgbouncer 1.12.0 sha256:1b3c6564376cafa0da98df3520f0e932bb2aebaf9a95ca5b9fa461e9eb7b273e
pgbouncer-exporter v0.1.3-gitlab git:6bfebdebe53d6e07815280bd9431bedbf5895c2a
pkg-config-lite 0.28-1 md5:61f05feb6bab0a6bbfab4b6e3b2f44b6
popt 1.16 md5:3743beefa3dd6247a73f8f7a32c14c33
postgres-exporter v0.8.0 git:3ecfd46549fa19006dd2399f79349cdf5a653ec8
postgresql 10.12 sha256:388f7f888c4fbcbdf424ec2bce52535195b426010b720af7bea767e23e594ae7
postgresql_new 11.7 sha256:324ae93a8846fbb6a25d562d271bc441ffa8794654c5b2839384834de220a313
postgresql_old 9.6.17 sha256:f6e1e32d32545f97c066f3c19f4d58dfab1205c01252cf85c5c92294ace1a0c2
prometheus v2.13.1 git:350f25eed339e5853e8d64ef19114429077d9018
python-docutils 0.13.1
python3 3.7.3 sha256:d62e3015f2f89c970ac52343976b406694931742fbde2fed8d1ce8ebb4e1f8ff
rb-readline master git:9fba246073f78831b7c7129c76cc07d8476a8892
redis 5.0.7 git:1f7d08b76dc83fd2aa43ee7fc04ab4ee01016628
redis-exporter v1.3.5 git:18080da36b0a97595d308cfb4bdf92eb1e396c44
registry v2.8.2-gitlab git:ae2e973db9365d6f8401ffb6bfc603f3cd76b111
remote-syslog 1.6.15
repmgr v3.3.2 git:0141bc2be70dbf7427f09fcc8ea2d3dfe60c1be5
repmgr_pg_10 v3.3.2 git:b691a1bd1063c083c205f344a028ea4c41fc146d
repmgr_pg_11 v3.3.2 git:b691a1bd1063c083c205f344a028ea4c41fc146d
rsync 3.1.3 md5:1581a588fde9d89f6bc6201e8129afaf
ruby 2.6.5 sha256:66976b716ecc1fd34f9b7c3c2b07bbd37631815377a2e3e85a5b194cfdcbed7d
rubygems 2.7.10
runit 2.1.2 md5:6c985fbfe3a34608eb3c53dc719172c4
unzip 6.0 sha256:036d96991646d0449ed0aa952e4fbe21b476ce994abc276e49d30e686708bd37
version-manifest 0.0.1
zlib v1.2.11 git:cacf7f1d4e3d44d871b605da3b647f07d718623f Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
Edited by Julian Paul Dasmarinas