Artifacts from different pipelines copied / mixed into same pipeline

Summary

tl;dr: A later stage in a pipleine sometimes receives artifacts from earlier stages from multiple/different pipelines of the same project if GIT_STRATEGY: none is set.

(Simplified) details:

We use Docker-Runners for our pipelines.

A pipeline consists of three steps/stages:

build: Builds a package and stores it as artifact in $CI_PROJECT_DIR/artifacts
test: Testing and linting
upload: Upload the artifacts from the build stage.

We have a master/main and a develop branch, but the problem also occurs if thereis only a single branch. Pipelines are triggered multiple times a day and are also scheduled to run every night.

Sometimes, the artifacts/ directory provided to the upload step contains packages created in the build step of two different pipelines (of the same project): the current pipeline and another, older pipeline.

It seems like this behavior is triggered by adding the variable GIT_STRATEGRY: 'none' to the upload step.

I analyzed the Docker volumes created by the pipelines as well as the job information from the GitLab API:

The artifacts created in each build step are clean (not mixed with other artifacts) and Job data retrieved via the GitLab API also indicates that only the expected files are in the build job's artifacts.
The artifacts stored in GitLab (.../gitlab/gitlab-rails/shared/artifacts) have the same "hash directory", but that directory contains artifacts from a lot of other jobs, too. The files in the artifacts.zip for the correspoinding jobs look okay.
When the volume for the upload step is created on the GitLab-Runner, GitLab first copies the artifacts of the wrong/older pipeline into it and than the artifacts of the current/correct pipeline. Duplicate files are overwriten, non-duplicate files are kept.

In our case, this results in packages from the master branch being uploaded into the staging repository and packages from the develop branch being upload to the stable repository. 💥

Steps to reproduce

Create a new Project

Add a .gitlab-ci.yml:

stages:
  - build
  - upload

default:
  image: ubuntu

build:
  stage: 'build'
  script:
    - 'mkdir artifacts'
    - 'echo $CI_JOB_ID > artifacts/$CI_JOB_ID'
    - 'echo $CI_PIPELINE_ID > artifacts/$CI_PIPELINE_ID'
  artifacts:
    expire_in: '7 days'
    paths:
      - 'artifacts'

upload:
  stage: 'upload'
  variables:
    GIT_STRATEGY: 'none'
  script:
    - 'ls -l artifacts/'
    - '[[ $(ls artifacts | wc -l) == 2 ]]'

Create a pipeline schedule that runs * * * * * or manually run a few pipelines.

Wait until one pipeline fails during the upload step:

$ ls -l artifacts/
total 16
-rw-r--r-- 1 root root 8 Apr  3 12:44 1749055
-rw-r--r-- 1 root root 8 Apr  3 12:44 1749059
-rw-r--r-- 1 root root 7 Apr  3 12:44 519592
-rw-r--r-- 1 root root 7 Apr  3 12:44 519594
$ [[ $(ls artifacts | wc -l) == 2 ]]

Example Project

https://gitlab.com/sscherfke/pipeline-artifacts-injection-issue

I could not reproduce the issue there, though.

What is the current bug behavior?

Artifacts of a prior pipeline of the same project arte mixted into the artifacts of the current pipleine.

Effectively it looks like this:

rsync old-pipeline-build/artifacts current-pipeline-upload/artifacts
rsync current-pipeline-build/artifacts current-pipeline-upload/artifacts

What is the expected correct behavior?

Only artifacts created in the current pipeline are available in the current pipeline.

Relevant logs and/or screenshots

–

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.3
Redis Version:  6.0.10
Git Version:    2.29.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.10.0
Revision:       5eafdaf7b07
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.5
URL:            https://git.services.xyz
HTTP Clone URL: https://git.services.xyz/some-group/some-project.git
SSH Clone URL:  git@git.services.xyz:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml

GitLab Shell
Version:        13.17.0
Repository storage paths:
- default:      /data/git/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.17.0 ? ... OK (13.17.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Checking Reply by email ...

IMAP server credentials are correct? ... Checking gitlab-incoming yes Init.d configured correctly? ... skipped MailRoom running? ... skipped

Checking Reply by email ... Finished

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 20/5 ... yes ... (all yes)

Redis version >= 4.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.29.0 ? ... yes (2.29.0) Git user has default SSH configuration? ... yes Active users: ... 143 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes

Checking GitLab App ... Finished