Registry push fails with large files
Summary
I am unable to push images to the private registry that are greater than 16 GB using the official gitlab/gitlab-ce docker image.
Eventually (after several retries), I see the error
received unexpected HTTP status: 502 Bad Gateway
Steps to reproduce
For brevity, I will not describe how to set up GitLab CE using the docker image, but I will reference the following two URLs with information about how to get HTTPS + the GitLab Container Registry to work.
https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https
https://docs.gitlab.com/ce/administration/container_registry.html
At first, I noticed that the push failed after I tried pushing a 24GB image. Then I was able to reliably demonstrate that it was failing for images >= 16 GB in size using the following shell script.
SIZE_GB=1
FAILURE=0
while [ $FAILURE -eq 0 ]; do
echo "========== TRYING WITH ${SIZE_GB} GB ========"
true \
&& docker build --build-arg SIZE_GB=${SIZE_GB} -t foo.bar.com:4567/namespace/project . \
&& docker push foo.bar.com:4567/namespace/project
FAILURE=$?
if [ $FAILURE -eq 0 ]; then
SIZE_GB=$((2*SIZE_GB))
fi
done
With the local Dockerfile shown below.
FROM ubuntu
ARG SIZE_GB=1
RUN head -c ${SIZE_GB}G < /dev/urandom > /hugefile
What is the current bug behavior?
Docker push completely uploads the >= 16GB image a few (3?) times, but retries each time directly after it has finished uploading. After the last retry, an HTTP error 502 is reported.
========== TRYING WITH 16 GB ========
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM ubuntu
---> ccc7a11d65b1
Step 2/3 : ARG SIZE_GB=1
---> Using cache
---> f3d89ffc7662
Step 3/3 : RUN head -c ${SIZE_GB}G < /dev/urandom > /hugefile
---> Running in 141e19fd848a
---> 62f9c0b85448
Successfully built 62f9c0b85448
Successfully tagged foo.bar.com:4567/namespace/project:latest
The push refers to a repository [foo.bar.com:4567/namespace/project]
f7a8d5272a0c: Pushing [==================================================>] 17.18GB
a09947e71dc0: Layer already exists
9c42c2077cde: Layer already exists
625c7a2a783b: Layer already exists
25e0901a71b8: Layer already exists
8aa4fcad5eeb: Layer already exists
received unexpected HTTP status: 502 Bad Gateway
I'm also seeing some strange output in /var/log/gitlab/registry/current. It's confusing, because I've definitely already authenticated to the registry via docker login
.
2017-10-02_15:38:35.18001 time="2017-10-02T15:38:35.145586673Z" level=warning msg="error authorizing context: invalid token" environment=production go.version=go1.8.1 http.request.host="foo.bar.com:4567" http.request.id=a0ec7adf-1726-4c24-8504-8daf8d4b0d74 http.request.method=PATCH http.request.remoteaddr=192.168.128.166 http.request.uri="/v2/namespace/project/blobs/uploads/d0fc2903-b46f-4086-b934-bba310c8fe54?_state=Gt9R4yLFUOPIryPS0L4WosrxE_fwtrB9l0Ye-5t-U6Z7Ik5hbWUiOiJwZXJ2aWNlcy1jbG9zZWQtc291cmNlL2ltYWdlLWJ1aWxkZXIiLCJVVUlEIjoiZDBmYzI5MDMtYjQ2Zi00MDg2LWI5MzQtYmJhMzEwYzhmZTU0IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTEwLTAyVDE1OjMwOjA0LjA2MjY1MjcwM1oifQ%3D%3D" http.request.useragent="docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87617 kernel/4.8.13-1-ARCH os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))" instance.id=47e0a144-d065-4ed9-8e7d-d61a5d4cd4a5 service=registry vars.name="namespace/project" vars.uuid=d0fc2903-b46f-4086-b934-bba310c8fe54 version=v2.6.1-1-gdd544a8
2017-10-02_15:38:35.18003 127.0.0.1 - - [02/Oct/2017:15:38:35 +0000] "PATCH /v2/namespace/project/blobs/uploads/d0fc2903-b46f-4086-b934-bba310c8fe54?_state=Gt9R4yLFUOPIryPS0L4WosrxE_fwtrB9l0Ye-5t-U6Z7Ik5hbWUiOiJwZXJ2aWNlcy1jbG9zZWQtc291cmNlL2ltYWdlLWJ1aWxkZXIiLCJVVUlEIjoiZDBmYzI5MDMtYjQ2Zi00MDg2LWI5MzQtYmJhMzEwYzhmZTU0IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTEwLTAyVDE1OjMwOjA0LjA2MjY1MjcwM1oifQ%3D%3D HTTP/1.0" 401 274 "" "docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87617 kernel/4.8.13-1-ARCH os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))"
2017-10-02_15:38:55.45318 time="2017-10-02T15:38:55.453148173Z" level=info msg="response completed" environment=production go.version=go1.8.1 http.request.host="foo.bar.com:4567" http.request.id=998deb51-2798-4038-843e-59db50864f07 http.request.method=POST http.request.remoteaddr=192.168.128.166 http.request.uri="/v2/namespace/project/blobs/uploads/" http.request.useragent="docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87617 kernel/4.8.13-1-ARCH os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))" http.response.duration=128.682882ms http.response.status=202 http.response.written=0 instance.id=47e0a144-d065-4ed9-8e7d-d61a5d4cd4a5 service=registry version=v2.6.1-1-gdd544a8
2017-10-02_15:38:55.45320 127.0.0.1 - - [02/Oct/2017:15:38:55 +0000] "POST /v2/namespace/project/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87617 kernel/4.8.13-1-ARCH os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))"
2017-10-02_15:47:26.28761 time="2017-10-02T15:47:26.287562231Z" level=info msg="token not to be used after 2017-10-02 15:44:55 +0000 UTC - currently 2017-10-02 15:47:26.287277108 +0000 UTC"
2017-10-02_15:47:26.30237 time="2017-10-02T15:47:26.287638374Z" level=warning msg="error authorizing context: invalid token" environment=production go.version=go1.8.1 http.request.host="foo.bar.com:4567" http.request.id=b2395edd-5c24-413e-90e8-3eee621422c6 http.request.method=PATCH http.request.remoteaddr=192.168.128.166 http.request.uri="/v2/namespace/project/blobs/uploads/559d02ce-75b3-4a05-9693-18314dabda50?_state=9Fx7ZHW4W5AGQPxvDYSpuKmNt48m1aKwMrOmTd-CoUd7Ik5hbWUiOiJwZXJ2aWNlcy1jbG9zZWQtc291cmNlL2ltYWdlLWJ1aWxkZXIiLCJVVUlEIjoiNTU5ZDAyY2UtNzViMy00YTA1LTk2OTMtMTgzMTRkYWJkYTUwIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTEwLTAyVDE1OjM4OjU1LjM3NjA5MDI1MloifQ%3D%3D" http.request.useragent="docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87617 kernel/4.8.13-1-ARCH os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))" instance.id=47e0a144-d065-4ed9-8e7d-d61a5d4cd4a5 service=registry vars.name="namespace/project" vars.uuid=559d02ce-75b3-4a05-9693-18314dabda50 version=v2.6.1-1-gdd544a8
2017-10-02_15:47:26.30238 127.0.0.1 - - [02/Oct/2017:15:47:26 +0000] "PATCH /v2/namespace/project/blobs/uploads/559d02ce-75b3-4a05-9693-18314dabda50?_state=9Fx7ZHW4W5AGQPxvDYSpuKmNt48m1aKwMrOmTd-CoUd7Ik5hbWUiOiJwZXJ2aWNlcy1jbG9zZWQtc291cmNlL2ltYWdlLWJ1aWxkZXIiLCJVVUlEIjoiNTU5ZDAyY2UtNzViMy00YTA1LTk2OTMtMTgzMTRkYWJkYTUwIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTEwLTAyVDE1OjM4OjU1LjM3NjA5MDI1MloifQ%3D%3D HTTP/1.0" 401 274 "" "docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87617 kernel/4.8.13-1-ARCH os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))"
What is the expected correct behavior?
I expect that the push should succeed and that docker should report
latest: digest: sha256:... size: ...
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 12.0.0 Redis Version: 3.2.5 Git Version: 2.13.5 Sidekiq Version:5.0.4 Go Version: unknownGitLab information Version: 9.5.4 Revision: fbffc27 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://foo.bar.com HTTP Clone URL: https://foo.bar.com/some-group/some-project.git SSH Clone URL: git@roxy.pv:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.8.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 5.8.0 ? ... OK (5.8.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 9/6 ... ok 9/7 ... ok 10/8 ... ok 2/10 ... ok 10/11 ... ok 2/12 ... ok 2/13 ... ok 10/14 ... ok 2/15 ... ok 2/17 ... ok 9/21 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 9/6 ... yes 9/7 ... yes 10/8 ... yes 2/10 ... yes 10/11 ... yes 2/12 ... yes 2/13 ... yes 10/14 ... yes 2/15 ... yes 2/17 ... yes 9/21 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.3) Git version >= 2.7.3 ? ... yes (2.13.5) Active users: ... 3
Checking GitLab ... Finished