Docker swarm / registry.gitlab.com - workers unable to pull?

Summary

I'm trying to use my private project's gitlab registry images in Docker swarm. While my master node can pull images, worker nodes cannot.

Steps to reproduce

Running on Centos Atomic 7.20170117 with docker 1.12.3, build 71b8cd3.

On the master host:

docker login -u ... -p .... registry.gitlab.com
docker service create --name something --with-registry-auth registry.gitlab.com/username/project:latest

Wait a few mins for the child node to try pulling. journalctl -u docker on the child node shows pull errors.

What is the current bug behavior?

The registry token does not work and the child is unable to pull the image.

What is the expected correct behavior?

The registry auth tokens passed from the master to the child should work correctly and the child should pull and run the container.

Relevant logs and/or screenshots

From using my japlanning-itins repo as the example.

Feb 11 11:01:19 worker-01 dockerd-latest[7389]: time="2017-02-11T11:01:19.861102008Z" level=error msg="Error trying v2 registry: Get https://registry.gitlab.com/v2/samjarrett/japlanning-itins/manifests/sha256:94477b6151a20e214bb65e43d14fa3e713dd9ed1c2ecfcaa99325ec1626c19b3: denied: access forbidden"
Feb 11 11:01:19 worker-01 dockerd-latest[7389]: time="2017-02-11T11:01:19.862112821Z" level=error msg="Attempting next endpoint for pull after error: Get https://registry.gitlab.com/v2/samjarrett/japlanning-itins/manifests/sha256:94477b6151a20e214bb65e43d14fa3e713dd9ed1c2ecfcaa99325ec1626c19b3: denied: access forbidden"
Feb 11 11:01:19 worker-01 dockerd-latest[7389]: time="2017-02-11T11:01:19.862149184Z" level=error msg="pulling image failed" error="Get https://registry.gitlab.com/v2/samjarrett/japlanning-itins/manifests/sha256:94477b6151a20e214bb65e43d14fa3e713dd9ed1c2ecfcaa99325ec1626c19b3: denied: access forbidden" module=taskmanager task.id=51gartc415305dn6b2f8ji2w6
Feb 11 11:01:19 worker-01 dockerd-latest[7389]: time="2017-02-11T11:01:19.862968673Z" level=error msg="fatal task error" error="No such image: registry.gitlab.com/samjarrett/japlanning-itins@sha256:94477b6151a20e214bb65e43d14fa3e713dd9ed1c2ecfcaa99325ec1626c19b3" module=taskmanager task.id=51gartc415305dn6b2f8ji2w6

Output of checks

This bug happens on GitLab.com

Edited Sep 02, 2020 by 🤖 GitLab Bot 🤖
Assignee Loading
Time tracking Loading