Cannot run Kaniko builds since GitLab Runner 16.6.0
Summary
Since we upgraded the runner to 16.6.0 we have this error each time.
error building image: error building stage: failed to get filesystem from image: error removing sbin to make way for new symlink: unlinkat //sbin/docker-init: device or resource busyreverting to 16.5.0 fixes the issue
The job is a container build using kaniko, the snipplet comes from the gitlab documentation
Steps to reproduce
# Taken from https://gitlab.com/gitlab-org/gitlab-foss/-/commit/e916a9880b4ea8c7cd4d521b4260c4732e8e2978
build-container:
rules:
- *do_not_run_on_push_except_on_main_branch
- when: always
stage: build
variables:
# Additional options for Kaniko executor.
# For more details see https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#additional-flags
KANIKO_ARGS: ""
KANIKO_BUILD_CONTEXT: $CI_PROJECT_DIR
image:
# For latest releases see https://github.com/GoogleContainerTools/kaniko/releases
# Only debug/*-debug versions of the Kaniko image are known to work within Gitlab CI
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
# if the user provide IMAGE_TAG then use it, else build the image tag using the default logic.
# Default logic
# Compose docker tag name
# Git Branch/Tag to Docker Image Tag Mapping
# * Default Branch: main -> latest
# * Branch: feature/my-feature -> branch-feature-my-feature
# * Tag: v1.0.0/beta2 -> v1.0.0-beta2
- |
if [ -z ${IMAGE_TAG+x} ]; then
if [ "$CI_COMMIT_REF_NAME" = $CI_DEFAULT_BRANCH ]; then
VERSION="latest"
elif [ -n "$CI_COMMIT_TAG" ];then
NOSLASH=$(echo "$CI_COMMIT_TAG" | tr -s / - )
SANITIZED="${NOSLASH//[^a-zA-Z0-9\-\.]/}"
VERSION="$SANITIZED"
else \
NOSLASH=$(echo "$CI_COMMIT_REF_NAME" | tr -s / - )
SANITIZED="${NOSLASH//[^a-zA-Z0-9\-]/}"
VERSION="branch-$SANITIZED"
fi
export IMAGE_TAG=$CI_REGISTRY_IMAGE:$VERSION
fi
- mkdir -p /kaniko/.docker
# Write credentials to access Gitlab Container Registry within the runner/ci
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
# Build and push the container. To disable push add --no-push
- DOCKERFILE_PATH=${DOCKERFILE_PATH:-"$KANIKO_BUILD_CONTEXT/Dockerfile"}
- /kaniko/executor --context $KANIKO_BUILD_CONTEXT --dockerfile $DOCKERFILE_PATH --destination $IMAGE_TAG $KANIKO_ARGS --cache=true
needs: ["check"]Docker file
FROM python:3.11-slim
WORKDIR /app
COPY . .
RUN mkdir venv
RUN /bin/bash -c "echo ./venv > .venv"
RUN /bin/bash -c "pip install pipenv"
RUN /usr/local/bin/pipenv sync
RUN export PATH=venv/bin:$PATH
EXPOSE 5000
CMD ["venv/bin/gunicorn", "--preload", "-b 0.0.0.0:8000", "--worker-class=eventlet", "wsgi:app"]Actual behavior
build fails
Expected behavior
build working as previously
Relevant logs and/or screenshots
job log
Running with gitlab-runner 16.6.0 (3046fee8)
on gitlab-runner-2 9R2E5zRm, system ID: s_c4b7f727d53b
Preparing the "docker" executor 00:01
Using Docker executor with image gcr.io/kaniko-project/executor:debug ...
Pulling docker image gcr.io/kaniko-project/executor:debug ...
Using docker image sha256:989556a3003989f120bc421ae6c837e6a3b8366e44b1b6947061929b93d9f295 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:b6a520e10252cfbb1898cac88d05f4c7c838cf615d1a588ec63a0d415db6c4c0 ...
Preparing environment 00:01
Running on runner-9r2e5zrm-project-102-concurrent-0 via gitlab-runner-2...
Getting source from Git repository 00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /tmp/builds/9R2E5zRm/0/ai-group/gaia/.git/
Checking out a3c9fab9 as detached HEAD (ref is refs/merge-requests/7/head)...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:03
Using docker image sha256:989556a3003989f120bc421ae6c837e6a3b8366e44b1b6947061929b93d9f295 for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:b6a520e10252cfbb1898cac88d05f4c7c838cf615d1a588ec63a0d415db6c4c0 ...
$ if [ -z ${IMAGE_TAG+x} ]; then # collapsed multi-line command
$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
$ DOCKERFILE_PATH=${DOCKERFILE_PATH:-"$KANIKO_BUILD_CONTEXT/Dockerfile"}
$ /kaniko/executor --context $KANIKO_BUILD_CONTEXT --dockerfile $DOCKERFILE_PATH --destination $IMAGE_TAG $KANIKO_ARGS --cache=true
INFO[0001] Using dockerignore file: /tmp/builds/9R2E5zRm/0/ai-group/gaia/.dockerignore
INFO[0001] Retrieving image manifest python:3.11-slim
INFO[0001] Retrieving image python:3.11-slim from registry index.docker.io
INFO[0002] Retrieving image manifest python:3.11-slim
INFO[0002] Returning cached image manifest
INFO[0002] Built cross stage deps: map[]
INFO[0002] Retrieving image manifest python:3.11-slim
INFO[0002] Returning cached image manifest
INFO[0002] Retrieving image manifest python:3.11-slim
INFO[0002] Returning cached image manifest
INFO[0002] Executing 0 build triggers
INFO[0002] Building stage 'python:3.11-slim' [idx: '0', base-idx: '-1']
INFO[0002] Checking for cached layer gitlab.xxx.com:5050/ai-group/gaia/cache:144318598a967e9e4081b51439f38cd68f910f76a8fd173fc0939d4e6ca98c91...
INFO[0002] No cached layer found for cmd RUN mkdir venv
INFO[0002] Cmd: EXPOSE
INFO[0002] Adding exposed port: 5000/tcp
INFO[0002] Unpacking rootfs as cmd COPY . . requires it.
error building image: error building stage: failed to get filesystem from image: error removing sbin to make way for new symlink: unlinkat //sbin/docker-init: device or resource busy
Cleaning up project directory and file based variablesEnvironment description
own dedicated gitlab installation just upgraded in 16.6.0
config.toml contents
root@gitlab-runner-2:~# cat /etc/gitlab-runner/config.toml
concurrent = 20
check_interval = 5
listen_address = "xxxxxxxxxx:9252"
[session_server]
session_timeout = 1800
listen_address = "xxxxxxxxxxx:8093"
[[runners]]
name = "gitlab-runner-2"
url = "https://gitlab.alcmeon.com"
token = "xxxxxxxxxxxxxx"
executor = "docker"
builds_dir = "/tmp/builds"
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.xxxxxxx"
AccessKey = "xxxxxxx"
SecretKey = "xxxxxx"
BucketName = "gitlab-runner-cache"
BucketLocation = "GRA"
[runners.docker]
tls_verify = false
image = "ubuntu:18.04"
# privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/tmp/builds:/tmp/builds", "/var/run/docker.sock:/var/run/docker.sock"]
# pull_policy = ["if-not-present"]
shm_size = 0
Used GitLab Runner version
root@gitlab-runner-2:~# gitlab-runner --version
Version: 16.6.0
Git revision: 3046fee8
Git branch: 16-6-stable
GO version: go1.20.10
Built: 2023-11-16T21:00:26+0000
OS/Arch: linux/amd64Possible fixes
Reverting runner to 16.5.0