Skip to content

Pushing docker image to gitlab registry fails after 8.10 update

This breaks my company's workflow!

Summary

Pushing docker image during CI fails after 8.10 update. It worked fine before.

Steps to reproduce

In my .gitlab-ci.yml file, I have:

- docker info
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $REGISTRY
- docker push $MY_REGISTRY/my-team/my-project

When the build runs, I get:

$ docker info
Containers: 97
 Running: 0
 Paused: 0
 Stopped: 97
Images: 547
Server Version: 1.11.2
Storage Driver: devicemapper
 Pool Name: docker-253:0-17847758-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 6.548 GB
 Data Space Total: 107.4 GB
 Data Space Available: 85.81 GB
 Metadata Space Used: 36.1 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.111 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.703 GiB
Name: git.company-name.com
ID: ZIDB:VMNS:2LTS:JH7G:TPB4:E5ZU:KIJ4:OCKI:AT5B:35NS:5ARY:QRMY
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $REGISTRY
Login Succeeded
$ docker push $MY_REGISTRY/my-team/my-project
The push refers to a repository [git.company-name.com:4567/my-team/my-project]
9bd2df9d9ac2: Preparing
d39154d08267: Preparing
14bf145d0917: Preparing
82c27186c092: Preparing
5f70bf18a086: Preparing
5f70bf18a086: Preparing
5f70bf18a086: Preparing
5f70bf18a086: Preparing
5f70bf18a086: Preparing
5f70bf18a086: Preparing
5f70bf18a086: Preparing
5f70bf18a086: Preparing
c2bff8368aa9: Preparing
335a9bc21e98: Preparing
5f70bf18a086: Preparing
5f70bf18a086: Preparing
8b7f6df7615d: Preparing
5f70bf18a086: Preparing
947564b0c8d9: Preparing
5f70bf18a086: Preparing
5f70bf18a086: Preparing
335a9bc21e98: Waiting
8b7f6df7615d: Waiting
947564b0c8d9: Waiting
c2bff8368aa9: Waiting
82c27186c092: Layer already exists
5f70bf18a086: Layer already exists
c2bff8368aa9: Layer already exists
335a9bc21e98: Layer already exists
14bf145d0917: Layer already exists
9bd2df9d9ac2: Retrying in 5 seconds
8b7f6df7615d: Layer already exists
947564b0c8d9: Layer already exists
d39154d08267: Retrying in 5 seconds
9bd2df9d9ac2: Retrying in 4 seconds
d39154d08267: Retrying in 4 seconds
9bd2df9d9ac2: Retrying in 3 seconds
d39154d08267: Retrying in 3 seconds
9bd2df9d9ac2: Retrying in 2 seconds
d39154d08267: Retrying in 2 seconds
9bd2df9d9ac2: Retrying in 1 second
d39154d08267: Retrying in 1 second
9bd2df9d9ac2: Retrying in 10 seconds
d39154d08267: Retrying in 10 seconds
9bd2df9d9ac2: Retrying in 9 seconds
d39154d08267: Retrying in 9 seconds
9bd2df9d9ac2: Retrying in 8 seconds
d39154d08267: Retrying in 8 seconds
9bd2df9d9ac2: Retrying in 7 seconds
d39154d08267: Retrying in 7 seconds
9bd2df9d9ac2: Retrying in 6 seconds
d39154d08267: Retrying in 6 seconds
9bd2df9d9ac2: Retrying in 5 seconds
d39154d08267: Retrying in 5 seconds
9bd2df9d9ac2: Retrying in 4 seconds
d39154d08267: Retrying in 4 seconds
9bd2df9d9ac2: Retrying in 3 seconds
d39154d08267: Retrying in 3 seconds
9bd2df9d9ac2: Retrying in 2 seconds
d39154d08267: Retrying in 2 seconds
9bd2df9d9ac2: Retrying in 1 second
d39154d08267: Retrying in 1 second
9bd2df9d9ac2: Retrying in 15 seconds
d39154d08267: Retrying in 15 seconds
9bd2df9d9ac2: Retrying in 14 seconds
d39154d08267: Retrying in 14 seconds
9bd2df9d9ac2: Retrying in 13 seconds
d39154d08267: Retrying in 13 seconds
9bd2df9d9ac2: Retrying in 12 seconds
d39154d08267: Retrying in 12 seconds
9bd2df9d9ac2: Retrying in 11 seconds
d39154d08267: Retrying in 11 seconds
9bd2df9d9ac2: Retrying in 10 seconds
d39154d08267: Retrying in 10 seconds
9bd2df9d9ac2: Retrying in 9 seconds
d39154d08267: Retrying in 9 seconds
9bd2df9d9ac2: Retrying in 8 seconds
d39154d08267: Retrying in 8 seconds
9bd2df9d9ac2: Retrying in 7 seconds
d39154d08267: Retrying in 7 seconds
9bd2df9d9ac2: Retrying in 6 seconds
d39154d08267: Retrying in 6 seconds
9bd2df9d9ac2: Retrying in 5 seconds
d39154d08267: Retrying in 5 seconds
9bd2df9d9ac2: Retrying in 4 seconds
d39154d08267: Retrying in 4 seconds
9bd2df9d9ac2: Retrying in 3 seconds
d39154d08267: Retrying in 3 seconds
9bd2df9d9ac2: Retrying in 2 seconds
d39154d08267: Retrying in 2 seconds
9bd2df9d9ac2: Retrying in 1 second
d39154d08267: Retrying in 1 second
9bd2df9d9ac2: Retrying in 20 seconds
d39154d08267: Retrying in 20 seconds
9bd2df9d9ac2: Retrying in 19 seconds
d39154d08267: Retrying in 19 seconds
9bd2df9d9ac2: Retrying in 18 seconds
d39154d08267: Retrying in 18 seconds
9bd2df9d9ac2: Retrying in 17 seconds
d39154d08267: Retrying in 17 seconds
9bd2df9d9ac2: Retrying in 16 seconds
d39154d08267: Retrying in 16 seconds
9bd2df9d9ac2: Retrying in 15 seconds
d39154d08267: Retrying in 15 seconds
9bd2df9d9ac2: Retrying in 14 seconds
d39154d08267: Retrying in 14 seconds
9bd2df9d9ac2: Retrying in 13 seconds
d39154d08267: Retrying in 13 seconds
9bd2df9d9ac2: Retrying in 12 seconds
d39154d08267: Retrying in 12 seconds
9bd2df9d9ac2: Retrying in 11 seconds
d39154d08267: Retrying in 11 seconds
9bd2df9d9ac2: Retrying in 10 seconds
d39154d08267: Retrying in 10 seconds
9bd2df9d9ac2: Retrying in 9 seconds
d39154d08267: Retrying in 9 seconds
9bd2df9d9ac2: Retrying in 8 seconds
d39154d08267: Retrying in 8 seconds
9bd2df9d9ac2: Retrying in 7 seconds
d39154d08267: Retrying in 7 seconds
9bd2df9d9ac2: Retrying in 6 seconds
d39154d08267: Retrying in 6 seconds
9bd2df9d9ac2: Retrying in 5 seconds
d39154d08267: Retrying in 5 seconds
9bd2df9d9ac2: Retrying in 4 seconds
d39154d08267: Retrying in 4 seconds
9bd2df9d9ac2: Retrying in 3 seconds
d39154d08267: Retrying in 3 seconds
9bd2df9d9ac2: Retrying in 2 seconds
d39154d08267: Retrying in 2 seconds
9bd2df9d9ac2: Retrying in 1 second
d39154d08267: Retrying in 1 second
blob upload unknown

ERROR: Build failed: exit status 1

Expected behavior

No errors when pushing image, which was behaviour previous to 8.10

Output of checks

Results of GitLab Application Check

Checking GitLab Shell ...

GitLab Shell version >= 3.2.0 ? ... OK (3.2.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...

(list of projects with ... ok)

Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Check directories and files:
	/var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...

(list of projects with ... yes)

Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.1.8)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 12

Checking GitLab ... Finished

Results of GitLab Environment Info

System information
System:		CentOS 7.2.1511
Current User:	git
Using RVM:	no
Ruby Version:	2.1.8p440
Gem Version:	2.5.1
Bundler Version:1.10.6
Rake Version:	10.5.0
Sidekiq Version:4.1.4

GitLab information
Version:	8.10.0
Revision:	9dec995
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://git.company-name.com
HTTP Clone URL:	https://git.company-name.com/some-group/some-project.git
SSH Clone URL:	git@git.company-name.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	3.2.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:		/opt/gitlab/embedded/bin/git

Possible fixes

None known