Missing /usr/bin/gitlab-runner-helper
In a python project CI/CD, I have a .gitlab-ci.yml:
- the build phase just builds the virtualenv, and stores it in the cache,
- the test phase pulls the virtualenv from the cache and launches the tests.
(see below for the snippet)
During the build phase, the logs show these messages:
- Missing /usr/bin/gitlab-runner-helper. Extracting cache is disabled.
- Missing /usr/bin/gitlab-runner-helper. Creating cache is disabled.
So, the cache is not created and therefore the test phase fails, as it has no virtualenv.
The exact same configuration is working correctly on another project.
I have read all the issues regarding "Missing gitlab-runner-helper", and the nasty DEBUG issue, but this seems not to be the case.
UPDATE: This IS actually still related to the DEBUG issue, so I am closing it. See last comment.
Is there a way to debug what's going on and know why the executor thinks that the gitlab-runner-helper is not there while it is really there?
- I am running gitlab-ce 12.7.6,
- the gitlab-runner is version 12.7.1, it was installed from the deb package on a Ubuntu 18.0.4
root@gitlab-runner-02:~# gitlab-runner --version Version: 12.7.1 Git revision: 003fe500 Git branch: 12-7-stable GO version: go1.13.5 Built: 2020-01-23T09:16:39+0000 OS/Arch: linux/amd64 - I can find a gitlab-runner-helper:x86_64-003fe500 image on the machine where the runner is executing
root@gitlab-runner-02:~# docker run --rm -it gitlab/gitlab-runner-helper:x86_64-003fe500 gitlab-runner-helper --version Version: 12.7.1 Git revision: 003fe500 Git branch: 12-7-stable GO version: go1.13.5 Built: 2020-01-23T08:32:11+0000 OS/Arch: linux/amd64
Following are some log excerpts and configuration files.
logs from the build phase
Running with gitlab-runner 12.7.1 (003fe500)
on Runner Two RtsxH2TV
Using Docker executor with image python:3.7-buster ...
Pulling docker image python:3.7-buster ...
Using docker image sha256:efdecc2e377a2438af1cf9e07286b5f7ee3f418c43b4bbb540b3752fdc0e008b for python:3.7-buster ...
Running on runner-RtsxH2TV-project-106-concurrent-0 via gitlab-runner-02...
00:02
Fetching changes with git depth set to 50...
00:02
Initialized empty Git repository in /builds/atoka/news/atoka-news/.git/
Created fresh repository.
From https://gitlab.depp.it/atoka/news/atoka-news
* [new ref] refs/pipelines/3372 -> refs/pipelines/3372
* [new branch] master -> origin/master
Checking out 5ab4eef0 as master...
Skipping Git submodules setup
Missing /usr/bin/gitlab-runner-helper. Extracting cache is disabled.
$ pip install -U pip virtualenv
Requirement already up-to-date: pip in /usr/local/lib/python3.7/site-packages (20.0.2)
...
Collecting distlib<1,>=0.3.0
Downloading distlib-0.3.0.zip (571 kB)
Building wheels for collected packages: distlib
...
Stored in directory: /builds/atoka/news/atoka-news/.pip-cache/wheels/eb/4e/d2/a903d4184fb49e4ac06474d65715b129aee13d69f7d227e78e
Successfully built distlib
Installing collected packages: appdirs, filelock, six, distlib, virtualenv
Successfully installed appdirs-1.4.3 distlib-0.3.0 filelock-3.0.12 six-1.14.0 virtualenv-20.0.4
$ virtualenv $VIRTUALENV_DIR
created virtual environment in 351ms CPython3Posix(dest=/builds/atoka/news/atoka-news/venv, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/root/.local/share/virtualenv/seed-v1 via=copy
$ source $VIRTUALENV_DIR/bin/activate
$ pip install -U pip-tools
Collecting pip-tools
Downloading pip_tools-4.5.0-py2.py3-none-any.whl (41 kB)
Collecting click>=7
Downloading Click-7.0-py2.py3-none-any.whl (81 kB)
Collecting six
Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
...
Successfully installed backcall-0.1.0 certifi-2019.11.28 cffi-1.14.0 chardet-3.0.4 coverage-5.0.3 cryptography-2.8 decorator-4.4.1 django-2.2.10 django-cors-headers-3.2.1 django-debug-toolbar-2.2 django-environ-0.4.5 django-extensions-2.2.8 django-model-utils-4.0.0 django-proxy-1.2.1 djangorestframework-3.11.0 djangorestframework-jwt-1.11.0 entrypoints-0.3 flake8-3.7.9 idna-2.8 ipython-7.12.0 ipython-genutils-0.2.0 jedi-0.16.0 josepy-1.3.0 markdown-3.2.1 mccabe-0.6.1 mistune-0.8.4 mozilla-django-oidc-1.2.3 parso-0.6.1 pexpect-4.8.0 pickleshare-0.7.5 prompt-toolkit-3.0.3 psycopg2-binary-2.8.4 ptyprocess-0.6.0 pycodestyle-2.5.0 pycparser-2.19 pyflakes-2.1.1 pygments-2.5.2 pyjwt-1.7.1 pyopenssl-19.1.0 pytz-2019.3 requests-2.22.0 sqlparse-0.3.0 traitlets-4.3.3 urllib3-1.25.8 wcwidth-0.1.8
$ pip freeze
...
urllib3==1.25.8
wcwidth==0.1.8
Missing /usr/bin/gitlab-runner-helper. Creating cache is disabled.
00:02
Job succeeded.gitlab-ci.yml
stages:
- build
- test
# Build: build virtualenv job
# -----------------------------------------------------------------------------
build virtualenv:
image: python:3.7-buster
stage: build
cache:
key: virtualenv-${CI_PROJECT_NAME}@${CI_COMMIT_REF_SLUG}
paths:
- ${CI_PROJECT_DIR}/.pip-cache
- ${CI_PROJECT_DIR}/venv
variables:
DATABASE_URL: "sqlite://:memory:"
script:
- pip install -U pip virtualenv
- virtualenv ${CI_PROJECT_DIR}/venv # Create virtual environment
- source ${CI_PROJECT_DIR}/venv/bin/activate # Activate virtual environment
- pip install -U pip-tools
- pip-sync requirements.txt # Install project requirements inside venv
- pip freeze # List installed packages in venv
except:
- schedules
# Test: django tests job
django tests:
image: python:3.7-buster
stage: test
variables:
DATABASE_URL: "sqlite://:memory:"
DJANGO_SETTINGS_MODULE: "config.settings"
cache:
key: virtualenv-${CI_PROJECT_NAME}@${CI_COMMIT_REF_SLUG}
policy: pull
paths:
- ${CI_PROJECT_DIR}/.pip-cache
- ${CI_PROJECT_DIR}/venv
before_script:
- source ${CI_PROJECT_DIR}/venv/bin/activate # Activate virtual environment
- pip freeze # List installed packages in virtualenv
script:
- coverage run manage.py test
- coverage report --omit="*/test*"
retry: 2
except:
- schedules/etc/gitlab-runner/config.toml
concurrent = 4
check_interval = 0
log_level = "debug"
[session_server]
listen_address = "0.0.0.0:8093"
advertise_address = "DOMAIN:8093"
session_timeout = 1800
[[runners]]
name = "My runner"
url = "URL"
token = "TOKEN"
executor = "docker"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Type = "s3"
Path = "s3-cache/"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
AccessKey = "ACCESS"
SecretKey = "SECRET"
BucketName = "NAME"
BucketLocation = "eu-central-1"Edited by Guglielmo Celata