Runner support for git fetch strategy on k8s breaks python venv caching
Summary
Prior to gitlab runner 13.12, $CI_PROJECT_DIR
would have the format /builds/<path/to/project>
, whereas now, the format seems to be /builds/<runner-token-prefix>/<parallel-job-idx>/<path/to/project>
. Some discussion for this change was here, in working out support for GIT_STRATEGY on k8s runners !2862 (comment 563791655).
Ultimately, the issue here is that if job-A creates an object in the cache, job-B can no longer depend on the absolute path being the same when the cache is extracted.
Steps to reproduce
My use case is that we cache python dependencies with the runner caching mechanisms; so
- Create a job that uses caches (our job is included below)
- Run the job
- In another job, attempt to load dependencies from the old cache
.gitlab-ci.yml
.ephemeral:
image: python:3.8
variables:
GET_POETRY: https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
POETRY_HOME: $CI_PROJECT_DIR/.cache/poetry
CACHE_FALLBACK_KEY: master
cache: &cache-defaults
key: $CI_COMMIT_REF_SLUG
policy: pull # Other jobs will override using <<: *cache-defaults if they need to push
paths:
# Cache folders only work for files within the project!
- .cache/poetry
- .cache/.poetry
- .cache/virtualenvs
before_script: &before-script
# Download poetry if it hasn't been cached already
- |
if [[ ! -r $POETRY_HOME/env ]]; then
mkdir -p $POETRY_HOME
curl -sSL $GET_POETRY -o $POETRY_HOME/get-poetry.py
POETRY_HOME=$POETRY_HOME python $POETRY_HOME/get-poetry.py --yes
fi
# Activate and configure poetry
- source $POETRY_HOME/env
- poetry config cache-dir $CI_PROJECT_DIR/.cache/.poetry
- poetry config virtualenvs.create true
- poetry config virtualenvs.in-project true
# Install python dependencies
- poetry install
Actual behavior
After extracting the cache, things have moved to different locations. Thus, this first poetry command, for-instance, fails:
$ source $POETRY_HOME/env
$ poetry config cache-dir $CI_PROJECT_DIR/.cache/.poetry
/bin/bash: line 157: poetry: command not found
Expected behavior
Obviously, we'd like to be able to use things extracted from the cache. So, after extracting the cache, tooling should be available and dependencies should already be installed.
$ source $POETRY_HOME/env
$ poetry config cache-dir $CI_PROJECT_DIR/.cache/.poetry
$ poetry config virtualenvs.create true
$ poetry config virtualenvs.in-project true
$ poetry install
Relevant logs and/or screenshots
successful job log
Running with gitlab-runner 13.12.0 (7a6612da)
on GKE Shared Runner XoCxmYd4
feature flags: FF_USE_FASTZIP:true, FF_GITLAB_REGISTRY_HELPER_IMAGE:true
Resolving secrets
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image python:3.8.5 ...
Preparing environment
Waiting for pod gitlab-runner/runner-xocxmyd4-project-76-concurrent-0fs2hc to be running, status is Pending
Running on runner-xocxmyd4-project-76-concurrent-0fs2hc via gitlab-runner-gitlab-runner-864454cbb6-mqlzf...
Getting source from Git repository
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/XoCxmYd4/0/path/to/project/.git/
Created fresh repository.
Checking out 25e4aa0e as refs/merge-requests/60/head...
Skipping Git submodules setup
Restoring cache
Checking cache for my-cache-key...
Downloading cache.zip from https://storage.googleapis.com/my-storage/gitlab-runner-cache/project/76/my-cache-key
WARNING: .venv/bin/python: chmod .venv/bin/python: no such file or directory (suppressing repeats)
Successfully extracted cache
Executing "step_script" stage of the job script
$ if [[ ! -r $POETRY_HOME/env ]]; then # collapsed multi-line command
$ export PATH="$POETRY_HOME/bin/:$PATH"
$ poetry config cache-dir $CI_PROJECT_DIR/.cache/.poetry
$ poetry config virtualenvs.create true
$ poetry config virtualenvs.in-project true
$ poetry install
Installing dependencies from lock file
No dependencies to install or update
Installing the current project: my_project (0.0.0+dynamic)
failed job log
This log is from a job after we applied our suggested fix for the `export`Running with gitlab-runner 13.12.0 (7a6612da)
on GKE Shared Runner XoCxmYd4
feature flags: FF_USE_FASTZIP:true, FF_GITLAB_REGISTRY_HELPER_IMAGE:true
Resolving secrets
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image python:3.8.5 ...
Preparing environment
Waiting for pod gitlab-runner/runner-xocxmyd4-project-76-concurrent-1vpw4k to be running, status is Pending
Running on runner-xocxmyd4-project-76-concurrent-1vpw4k via gitlab-runner-gitlab-runner-864454cbb6-mqlzf...
Getting source from Git repository
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/XoCxmYd4/1/path/to/project/.git/
Created fresh repository.
Checking out 25e4aa0e as refs/merge-requests/60/head...
Skipping Git submodules setup
Restoring cache
Checking cache for my-cache-key...
Downloading cache.zip from https://storage.googleapis.com/my-storage/gitlab-runner-cache/project/76/my-cache-key
WARNING: .venv/bin/python: chmod .venv/bin/python: no such file or directory (suppressing repeats)
Successfully extracted cache
Executing "step_script" stage of the job script
$ if [[ ! -r $POETRY_HOME/env ]]; then # collapsed multi-line command
$ export PATH="$POETRY_HOME/bin/:$PATH"
$ poetry config cache-dir $CI_PROJECT_DIR/.cache/.poetry
$ poetry config virtualenvs.create true
$ poetry config virtualenvs.in-project true
$ poetry install
Installing dependencies from lock file
No dependencies to install or update
Installing the current project: my_project (0.0.0+dynamic)
$ GIT_VERSION=$(poetry run python -c 'import my_project; print(my_project.__version__)')
$ poetry version $GIT_VERSION
Bumping version from 0.0.0+dynamic to 0.3.0.post6.dev0+25e4aa0
$ poetry run flake8 --config .flake8 tests/
FileNotFoundError
[Errno 2] No such file or directory: '/builds/XoCxmYd4/1/path/to/project/.venv/bin/flake8'
at /usr/local/lib/python3.8/os.py:591 in _execvpe
587│ argrest = (args,)
588│ env = environ
589│
590│ if path.dirname(file):
→ 591│ exec_func(file, *argrest)
592│ return
593│ saved_exc = None
594│ path_list = get_exec_path(env)
595│ if name != 'nt':
Cleaning up file based variables
ERROR: Job failed: command terminated with exit code 1
This is a self-hosted installation, we're using the gitlab-runner helm chart on k8s on gke. The version of the chart is 0.29.0 for gitlab-runner version 13.12.0
config.toml contents
concurrent = 10
check_interval = 30
log_level = "info"
listen_address = ':9252'
[[runners]]
[runners.feature_flags]
FF_GITLAB_REGISTRY_HELPER_IMAGE = true
FF_USE_FASTZIP = true
[runners.kubernetes]
image = "ubuntu:20.04"
[runners.cache]
Type = "gcs"
Path = "gitlab-runner-cache"
Shared = true
[runners.cache.gcs]
AccessId = "REDACTED"
PrivateKey = "REDACTED"
BucketName = "REDACTED"
Used GitLab Runner version
Possible fixes
Again, see here: !2862 (comment 563791655).
Really, this might be something to close as wontfix, but I thought I'd get my use-case down. I extracted two caches from before and after the 13.12 update, one working and one failing.
The diff of $CI_PROJECT_DIR/.cache/poetry/env
tells the story
- export PATH="/builds/path/to/my/project/.cache/poetry/bin:$PATH"
+ export PATH="/builds/XoCxmYd4/0/path/to/my/project/.cache/poetry/bin:$PATH"
As a user, I thought I could fix this easily by not depending on this env
script from one of our tools (poetry), and do the work ourselves with something like export PATH=$CI_BUILDS_DIR/.cache/poetry/bin:$PATH
, but there's more problems with cache contents not being extracted to the same absolute path again. I felt like making an issue for the visibility, and to share the use case, because there's probably lots of other jobs experiencing something like this.