Skip to content

Wrong signature pushed when multiple images are built in parallel

Describe the bug

When building multiple images using parallel:matrix, the same image signature is pushed for all images.

Expected behavior

Each image should have its own signature pushed to the registry.

Actual behavior

The same signature is pushed for all images.

Logs and/or screenshots

Logs of the docker-kaniko-build jobs:

INFO[0013] Pushing image to registry.xxx/docker-parallel-test/snapshot:1.29.0 
INFO[0014] Pushed xxx/docker-parallel-test/snapshot@sha256:2b8a0d220c4958b5af7d2f2a3ce4fdcf5e0f48659cbf4fcf7decec305b2b9a98

and

INFO[0015] Pushing image to xxx/docker-parallel-test/snapshot:1.30.0 
INFO[0016] Pushed registry.xxx/docker-parallel-test/snapshot@sha256:670a9b72bd75e843163b80eac9b48eea409c9d9278c8f49fc61593bed26cb8a1

tag 1.29.0 corresponds to sha256:2b8a0d220c4958b5af7d2f2a3ce4fdcf5e0f48659cbf4fcf7decec305b2b9a98

tag 1.30.0 corresponds to sha256:670a9b72bd75e843163b80eac9b48eea409c9d9278c8f49fc61593bed26cb8a1

Logs of the docker-publish jobs:

[INFO] Copying registry.xxx/docker-parallel-test/snapshot:1.29.0 to registry.xxx/docker-parallel-test:1.29.0...
$ skopeo copy --src-authfile "$BUILDTOOL_HOME/skopeo/.docker/src-config.json" --dest-authfile "$BUILDTOOL_HOME/skopeo/.docker/dest-config.json" ${DOCKER_PUBLISH_ARGS} "docker://$DOCKER_SNAPSHOT_IMAGE" "docker://$DOCKER_RELEASE_IMAGE"
Getting image source signatures
Copying blob sha256:4499ec5029785f40214ba8db2fc74a3b5ae45328a87f2e9b660c4885a59db2eb
Copying blob sha256:355df20be89879ae2832aba4190788a2db72f374ed25f09d071e72a999bcc63b
Copying blob sha256:fe07684b16b82247c3539ed86a65ff37a76138ec25d380bd80c869a1a4c73236
Copying blob sha256:1a386b6109839267143e5e8d9cb31cd0002fa2fb54f34cc1925a6ad4c7c8c415
Copying config sha256:8643f152bbf61372bdde9d67ff46b507f0dad73086f5169fdc6ec00eb5189a5e
Writing manifest to image destination
$ if [[ ${DOCKER_COSIGN_STRATEGY} == "onrelease" ]] || [[ ${DOCKER_COSIGN_STRATEGY} == "always" ]] # collapsed multi-line command
[INFO] Copying image signature to registry.xxx/docker-parallel-test:sha256-670a9b72bd75e843163b80eac9b48eea409c9d9278c8f49fc61593bed26cb8a1.sig...
Getting image source signatures
Copying blob sha256:fb400500de296bd87fa90c9506f4b6639c520198b5f045442ef3d34a7dbb1e34
Copying config sha256:21c0a7021dc7a2ec66ed25f4ba8850b0d81174032b92aed6b027b6aa9489bada
Writing manifest to image destination
[INFO] Copying image attestation to registry.xxx/docker-parallel-test:sha256-670a9b72bd75e843163b80eac9b48eea409c9d9278c8f49fc61593bed26cb8a1.att...

and

[INFO] Copying registry.xxx/docker-parallel-test/snapshot:1.30.0 to registry.xxx/docker-parallel-test:1.30.0...
$ skopeo copy --src-authfile "$BUILDTOOL_HOME/skopeo/.docker/src-config.json" --dest-authfile "$BUILDTOOL_HOME/skopeo/.docker/dest-config.json" ${DOCKER_PUBLISH_ARGS} "docker://$DOCKER_SNAPSHOT_IMAGE" "docker://$DOCKER_RELEASE_IMAGE"
Getting image source signatures
Copying blob sha256:ae6e2d0d5af2c435212b70ecd26c792f19c11e195ca0fafd7601d443ad9684f2
Copying blob sha256:f93dc828fc7a454d983392711c20b451a477310c515e7f8f7b535e60e76ba1f9
Copying blob sha256:fe07684b16b82247c3539ed86a65ff37a76138ec25d380bd80c869a1a4c73236
Copying blob sha256:afb33b2f779513328049822ba98b27b77b73bdef4ff11fbaa8c3c985dfcca95a
Copying config sha256:58c7c1b0aeeee931be0468ff448c7ebd32c3fab94c16ebf152f89ef742c09c78
Writing manifest to image destination
$ if [[ ${DOCKER_COSIGN_STRATEGY} == "onrelease" ]] || [[ ${DOCKER_COSIGN_STRATEGY} == "always" ]] # collapsed multi-line command
[INFO] Copying image signature to registry.xxx/docker-parallel-test:sha256-670a9b72bd75e843163b80eac9b48eea409c9d9278c8f49fc61593bed26cb8a1.sig...
Getting image source signatures
Copying blob sha256:fb400500de296bd87fa90c9506f4b6639c520198b5f045442ef3d34a7dbb1e34
Copying config sha256:21c0a7021dc7a2ec66ed25f4ba8850b0d81174032b92aed6b027b6aa9489bada
Writing manifest to image destination
[INFO] Copying image attestation to registry.xxx/docker-parallel-test:sha256-670a9b72bd75e843163b80eac9b48eea409c9d9278c8f49fc61593bed26cb8a1.att...

The signature and attestation related to sha256-670a9b72bd75e843163b80eac9b48eea409c9d9278c8f49fc61593bed26cb8a1 are attached to both image tags.

Context & Configuration

Link to a project, pipeline or job facing the bug: (please provide one if possible)

The issue was reproduced using:

  • Version of the template: 6.2.1
  • GitLab server(s): A self-managed server with CE
  • GitLab runner(s): Kubernetes

Here is the .gitlab-ci.yml file:

include:
  - project: 'to-be-continuous/docker'
    ref: '6.2.1'
    file: '/templates/gitlab-ci-docker.yml'

variables:
  DOCKER_PROD_PUBLISH_STRATEGY: "auto"
  DOCKER_TRIVY_ARGS: "--ignore-unfixed --vuln-type os --scanners vuln"
  DOCKER_BUILD_ARGS: "--build-arg KUBE_VERSION=${KUBE_VERSION_SHORT}.0 --build-arg HELM_VERSION=${HELM_VERSION} --cache-ttl=1h"
  DOCKER_SNAPSHOT_IMAGE: "$CI_REGISTRY_IMAGE/snapshot:${KUBE_VERSION_SHORT}.0"
  DOCKER_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE:${KUBE_VERSION_SHORT}.0"
  DOCKER_RELEASE_EXTRA_TAGS: "\\g<major>.\\g<minor>"
  DOCKER_COSIGN_STRATEGY: "onrelease"

.docker-base:
  parallel:
    matrix:
    - KUBE_VERSION_SHORT: "1.29"
      HELM_VERSION: "3.16.2"
    - KUBE_VERSION_SHORT: "1.30"
      HELM_VERSION: "3.16.2"

The problems comes from the exported docker.env dotenv artifact: a single artifact is exported even if multiple images are build in parallel. So in the next jobs, only the latest docker.env file is used for all parallel jobs.