Disabled registry on external Sidekiq nodes causes issues with CI_REGISTRY_IMAGE and CI_REGISTRY variables

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

A customer (internal) noticed issues with CI_REGISTRY_IMAGE and CI_REGISTRY variables in their pipelines after they changed their external sidekiq nodes to stop running the registry service. They configured their external registry according to Configure the Container Registry when using an external Sidekiq instructions and started seeing that CI_REGISTRY_IMAGE and CI_REGISTRY varialbe is empty when using it as part of another expandable variable on a trigger job for parent-child pipeline.

Steps to reproduce

  1. Deploy a GitLab multi-node instance with external sidekiq, make sure the instance has available runners to run jobs.
  2. Make sure that Container registry is not running on sidekiq node.
  3. Create a project and configure the following files for CI/CD pipelines 
    # .gitlab-ci.yml:
stages:
  - setup

pipeline:parent-child:
  stage: setup
  trigger:
    include: ci/standard.yml
    strategy: depend
  variables:
    IMAGE_TAG: "${CI_COMMIT_REF_SLUG}"
    DISTRO: ub20
    BASEIMAGE: ubuntu:focal
    CI_DEBUG_TRACE: "true"
    IMAGE: "${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}-build-${DISTRO}:${IMAGE_TAG}"


# ci/standard.yml
stages:
  - build-container

bld-container:
  stage: build-container
  image:
    name: "gcr.io/kaniko-project/executor:$KANIKO_VERSION"
    entrypoint: [""]
  variables:
    KANIKO_VERSION: v1.8.1-debug
  script:
    - echo "Pushing to $IMAGE"
  4. Run the pipeline and check the bld-container job output.

=> The part with ${CI_REGISTRY_IMAGE} will be empty and you should see the following output:

echo "Building $DOCKERFILE and pushing to $IMAGE"
Building ci/dockerfile-ub20 and pushing to /project_name-build-ub20:main

Note that this is only limited to expandable variables that are configured on a trigger job. Configuring the same variable on a normal job does not have this issue, the variable is correctly expanded.

Example Project

N/A

What is the current bug behavior?

${CI_REGISTRY_IMAGE} variable is expanded as an empty value when configured in a trigger job.

What is the expected correct behavior?

${CI_REGISTRY_IMAGE} variable should be expanded correctly value when used in in another variable in a trigger job.

Relevant logs and/or screenshots

Correct output of the job that ran on single-node instance:

$ echo "Building $DOCKERFILE and pushing to $IMAGE"
+ echo 'Building ci/dockerfile-ub20 and pushing to main.gitlab.egrechishkina.com:5050/bachmanity/zd444511/zd444511-build-ub20:main'
Building ci/dockerfile-ub20 and pushing to main.gitlab.egrechishkina.com:5050/bachmanity/zd444511/zd444511-build-ub20:main

Incorrect behaviour that is seen on multi-node setup with registry not running on sidekiq node:

+ echo 'Building ci/dockerfile-ub20 and pushing to /zd444511-build-ub20:main'
$ echo "Building $DOCKERFILE and pushing to $IMAGE"
Building ci/dockerfile-ub20 and pushing to /zd444511-build-ub20:main

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info System information System: Ubuntu 20.04 Proxy: no Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.4.18 Bundler Version:2.4.18 Rake Version: 13.0.6 Redis Version: 7.0.12 Sidekiq Version:6.5.7 Go Version: unknown

GitLab information Version: 16.3.0-ee Revision: 3dfd4e0b Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.11 URL: https://main.gitlab.egrechishkina.com HTTP Clone URL: https://main.gitlab.egrechishkina.com/some-group/some-project.git SSH Clone URL: git@main.gitlab.egrechishkina.com:some-group/some-project.git Elasticsearch: no Geo: yes Geo node: Primary Using LDAP: no Using Omniauth: yes Omniauth Providers: bitbucket

GitLab Shell Version: 14.26.0 Repository storages:

  • default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell

Gitaly

  • default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
  • default Version: 16.3.0
  • default Git Version: 2.41.0.gl1

Results of GitLab application Check

Expand for output related to the GitLab application check 
System information
System:		Ubuntu 20.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	3.0.6p216
Gem Version:	3.4.18
Bundler Version:2.4.18
Rake Version:	13.0.6
Redis Version:	7.0.12
Sidekiq Version:6.5.7
Go Version:	unknown

GitLab information
Version:	16.3.0-ee
Revision:	3dfd4e0bcc7
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	13.11
URL:		https://main.gitlab.egrechishkina.com
HTTP Clone URL:	https://main.gitlab.egrechishkina.com/some-group/some-project.git
SSH Clone URL:	git@main.gitlab.egrechishkina.com:some-group/some-project.git
Elasticsearch:	no
Geo:		yes
Geo node:	Primary
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: bitbucket


GitLab Shell
Version:	14.26.0
Repository storages:
  • default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell

Gitaly

  • default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
  • default Version: 16.3.0
  • default Git Version: 2.41.0.gl1 root@main-gl-instance:/home/khrechyshkina# sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.26.0 ? ... OK (14.26.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/1 ... yes 51/2 ... yes 51/3 ... yes 51/4 ... yes 51/5 ... yes 51/6 ... yes 51/7 ... yes 51/8 ... yes 51/9 ... yes 51/10 ... yes 51/11 ... yes 53/14 ... yes 53/15 ... yes 53/16 ... yes 54/17 ... yes 54/18 ... yes 54/19 ... yes 54/20 ... yes 54/21 ... yes 54/22 ... yes 54/23 ... yes 54/24 ... yes 54/25 ... yes 54/26 ... yes 55/27 ... yes 55/28 ... yes 55/29 ... yes 55/30 ... yes 56/31 ... yes 56/32 ... yes 56/33 ... yes 56/34 ... yes 56/35 ... yes 57/36 ... yes 57/37 ... yes 57/38 ... yes 57/39 ... yes 58/40 ... yes 58/41 ... yes 58/42 ... yes 66/43 ... yes 70/45 ... yes 70/46 ... yes 69/47 ... yes 69/48 ... yes 58/52 ... yes 58/53 ... yes 63/54 ... yes 66/55 ... yes 66/56 ... yes 66/57 ... yes 66/58 ... yes 66/59 ... yes 65/72 ... yes 66/77 ... yes 66/78 ... yes 69/81 ... yes 66/85 ... yes 66/86 ... yes 66/89 ... yes 66/90 ... yes 66/92 ... yes 66/97 ... yes 69/100 ... yes 66/101 ... yes 66/102 ... yes 66/103 ... yes 66/104 ... yes 66/107 ... yes 66/110 ... yes 69/111 ... yes 94/112 ... yes 66/113 ... yes 69/114 ... yes 91/115 ... yes 66/117 ... yes 66/118 ... yes 66/119 ... yes 66/121 ... yes 66/122 ... yes 66/124 ... yes 66/125 ... yes 66/126 ... yes 66/128 ... yes 66/129 ... yes 66/130 ... yes 66/131 ... yes 69/132 ... yes 66/133 ... yes 66/137 ... yes 91/138 ... yes 69/140 ... yes 69/141 ... yes 69/142 ... yes 66/143 ... yes 100/144 ... yes 102/145 ... yes 69/146 ... yes 66/147 ... yes 66/148 ... yes 66/149 ... yes 66/152 ... yes 69/153 ... yes 66/155 ... yes 66/156 ... yes 66/157 ... yes 66/159 ... yes 66/160 ... yes 66/161 ... yes 69/163 ... yes 60/164 ... yes 108/165 ... yes 66/166 ... yes 112/167 ... yes 111/168 ... yes 66/169 ... yes 66/171 ... yes 66/172 ... yes 66/173 ... yes 66/174 ... yes 66/175 ... yes 66/176 ... yes 66/177 ... yes 91/180 ... yes 120/184 ... yes 1/185 ... yes 1/186 ... yes 1/187 ... yes 66/189 ... yes 1/190 ... yes 1/191 ... yes 66/192 ... yes 130/193 ... yes 1/194 ... yes 272/195 ... yes 275/197 ... yes 1/198 ... yes 54/199 ... yes 54/200 ... yes 54/201 ... yes 272/202 ... yes 272/203 ... yes 1/204 ... yes 54/205 ... yes 272/206 ... yes 62/207 ... yes 54/208 ... yes 1/209 ... yes 54/210 ... yes 54/211 ... yes 54/212 ... yes 63/213 ... yes 59/214 ... yes 299/215 ... yes 66/216 ... yes 69/217 ... yes 69/219 ... yes 69/220 ... yes 69/221 ... yes 69/222 ... yes 69/223 ... yes 69/224 ... yes 310/225 ... yes 69/226 ... yes 69/227 ... yes 69/228 ... yes 69/229 ... yes 69/230 ... yes 69/231 ... yes 69/232 ... yes 69/233 ... yes 69/234 ... yes 69/235 ... yes 69/236 ... yes 69/237 ... yes 69/238 ... yes 69/239 ... yes 69/240 ... yes 69/241 ... yes 69/242 ... yes 69/243 ... yes 69/244 ... yes 69/245 ... yes 69/246 ... yes 69/247 ... yes 69/248 ... yes 69/249 ... yes 69/250 ... yes 69/251 ... yes 69/252 ... yes 69/253 ... yes 69/254 ... yes 69/255 ... yes 69/256 ... yes 69/257 ... yes 69/258 ... yes 69/259 ... yes 69/260 ... yes 69/261 ... yes 69/262 ... yes 69/263 ... yes 69/264 ... yes 69/265 ... yes 69/266 ... yes 69/267 ... yes 69/268 ... yes 69/269 ... yes 69/270 ... yes 69/271 ... yes 69/272 ... yes 69/273 ... yes 69/274 ... yes 69/275 ... yes 69/276 ... yes 69/277 ... yes 69/278 ... yes 69/279 ... yes 69/280 ... yes 69/281 ... yes 69/282 ... yes 69/283 ... yes 69/284 ... yes 69/285 ... yes 69/286 ... yes 69/287 ... yes 69/288 ... yes 69/289 ... yes 69/290 ... yes 69/291 ... yes 69/292 ... yes 69/293 ... yes 69/294 ... yes 69/295 ... yes 69/296 ... yes 69/297 ... yes 69/298 ... yes 69/299 ... yes 69/300 ... yes 69/301 ... yes 69/302 ... yes 69/303 ... yes 69/304 ... yes 69/305 ... yes 69/306 ... yes 63/307 ... yes 95/308 ... yes 54/309 ... yes 62/310 ... yes 397/311 ... yes 63/312 ... yes 57/313 ... yes 406/314 ... yes 408/315 ... yes 69/316 ... yes 69/317 ... yes 397/318 ... yes 53/319 ... yes 59/320 ... yes 63/321 ... yes 416/322 ... yes 66/324 ... yes 63/325 ... yes 63/326 ... yes 424/327 ... yes 63/328 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 3.0.6 ? ... yes (3.0.6) Git user has default SSH configuration? ... yes Active users: ... 10 Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled) All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)

Checking GitLab App ... Finished

Checking Geo ...

GitLab Geo is available ... GitLab Geo is enabled ... yes This machine's Geo node name matches a database record ... yes, found a primary node named "geo-primary-node" HTTP/HTTPS repository cloning is enabled ... yes Machine clock is synchronized ... yes Git user has default SSH configuration? ... yes OpenSSH configured to use AuthorizedKeysCommand ... yes GitLab configured to disable writing to authorized_keys file ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes

Checking Geo ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

I suspect this code to be responsible for not setting the variables correctly.

Edited by 🤖 GitLab Bot 🤖