gitlab-runner can't pull from gitlab registry

Summary

GitLab runner fails to pull an image on the GitLab registry with the following error:

Using Docker executor with image ctregistry.saphyre.net/packages/crm-api/php:da3ada5f ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image ctregistry.saphyre.net/packages/crm-api/php:da3ada5f ...
WARNING: Failed to pull image with policy "always": invalid character '<' looking for beginning of value (manager.go:203:1s)

Steps to reproduce

  • Have GitLab registry
  • Have gitlab/gitlab-runner:latest in docker with docker socket mounted
  • .gitlab-ci.yml as follows [1]
  • run pipeline
[1]: expand .gitlab-ci.yml 
variables:
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_TLS_VERIFY: 1
  DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  DOCKER_DRIVER: overlay2
  PHP_TEST_IMAGE: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/crm-api/php

stages:
  - build
  - test

image:
  name: docker:20.10.12

build:
  stage: build
  services:
    - name: docker:20.10.12-dind
  variables:
    DOCKER_BUILDKIT: 1
  script:
    - apk update
    - apk add docker-compose
    - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker pull ${PHP_TEST_IMAGE}:${CI_COMMIT_SHORT_SHA} || ./scripts/build_and_push.sh
  only:
    refs:
      - master@packages/crm-api
      - merge_requests
      - tags

codequality:
  stage: test
  image: "${PHP_TEST_IMAGE}:${CI_COMMIT_SHORT_SHA}"
  script:
    - apk add jq
    - composer install
    - ./scripts/run_quality_tests.sh
  artifacts:
    when: always
    reports:
      codequality: quality-report.json
  only:
    refs:
      - master@packages/crm-api
      - merge_requests
      - tags

Example Project

What is the current bug behavior?

  • The build job in the build stage pulls docker:20.10.12 and starts service docker:20.10.12-dind for a dind set-up
  • docker login successfully authenticates to $CI_REGISTRY using credentials in $CI_REGISTRY_USER and $CI_JOB_TOKEN
  • ./scripts/build_and_push.sh successfully builds the image and pushes it to the registry ($PHP_TEST_IMAGE)
  • The codequality job in the test stage tries to pull the image we just pushed but fails with the error as above

What is the expected correct behavior?

  • The codequality job in the test stage should be able to pull the image we just pushed

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.2.33
Rake Version:	13.0.6
Redis Version:	6.2.6
Sidekiq Version:6.4.0
Go Version:	unknown

GitLab information
Version:	14.9.1
Revision:	5b713e1e9a3
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.7
URL:		https://gitlab.saphyre.net
HTTP Clone URL:	https://gitlab.saphyre.net/some-group/some-project.git
SSH Clone URL:	git@gitlab.saphyre.net:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers: google_oauth2

GitLab Shell
Version:	13.24.0
Repository storage paths:
- default: 	/srv/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.24.0 ? ... OK (13.24.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Failed. Check bind_dn and password configuration values
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 3 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
3/1 ... yes
3/2 ... yes
2/3 ... yes
2/4 ... yes
5/5 ... yes
2/6 ... yes
2/9 ... yes
6/10 ... yes
6/11 ... yes
6/12 ... yes
6/13 ... yes
6/14 ... yes
2/15 ... yes
3/16 ... yes
2/17 ... yes
3/18 ... yes
2/19 ... yes
10/20 ... yes
2/21 ... yes
6/22 ... yes
5/23 ... yes
2/24 ... yes
5/25 ... yes
2/26 ... yes
3/28 ... yes
2/29 ... yes
10/30 ... yes
2/31 ... yes
10/32 ... yes
2/33 ... yes
11/34 ... yes
11/35 ... yes
11/36 ... yes
11/37 ... yes
11/38 ... yes
11/39 ... yes
6/41 ... yes
10/43 ... yes
6/44 ... yes
2/45 ... yes
10/47 ... yes
2/48 ... yes
6/49 ... yes
10/50 ... yes
2/51 ... yes
6/52 ... yes
11/53 ... yes
2/54 ... yes
6/55 ... yes
10/56 ... yes
2/57 ... yes
6/58 ... yes
10/59 ... yes
2/60 ... yes
6/61 ... yes
11/62 ... yes
2/63 ... yes
6/64 ... yes
6/65 ... yes
11/66 ... yes
2/67 ... yes
6/68 ... yes
10/69 ... yes
2/70 ... yes
10/71 ... yes
2/72 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 8
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes