Executing podman inside docker runner

Summary

I had some issue with space on one my runners and used this circumstance to also upgrade the runner.

The runner is configured as linux runner with docker executor.

After Upgraded from manual installation to debian poackage.

Version: 17.11.0 -> 18.6.1

After the upgrade the job I was running fails with:

Running command:
  podman build -f context/Containerfile -t gitlab.fb10.fh-dortmund.de:5005/fb10-it/ansible/gitlab-fb10/gitlab-ee:latest context
cannot clone: Operation not permitted
Error: cannot re-exec process

I try to build an ansible execution environment which uses podman to build and push the image to registry.

Steps to reproduce

I provide the masked runner config and gitlab-ci.yml for this job. Additional I provide a simple exection_environment.yml file which is used to build the ee. In previous runs with 17.11.0 the runner executed further than the failed command.

ee/execution-environment.yml

version: 3

images:
  base_image:
    name: registry.fedoraproject.org/fedora:42

dependencies:
  python_interpreter:
    package_system: python3
  ansible_core:
    package_pip: ansible-core
  ansible_runner:
    package_pip: ansible-runner
  system:
    - openssh-clients
    - sshpass

.gitlab-ci.yml

variables:
  UV_VERSION: "0.5"
  PYTHON_VERSION: "3.12"
  BASE_LAYER: bookworm-slim
  UV_LINK_MODE: copy
  IMAGE_TAG: $CI_REGISTRY_IMAGE/gitlab-ee:latest

stages:
  - build

build-and-deploy-ee:
  image: quay.io/podman/stable:latest
  stage: build
  variables:
    STORAGE_DRIVER: vfs
  before_script:
    - dnf install -y curl python3 python3-pip
    - curl -LsSf https://astral.sh/uv/install.sh | sh
    - source ~/.local/bin/env
    - uv --version
  script:
    - uv pip compile pyproject.toml --group ee -o ee/requirements.txt
    - cd ee
    - echo "Running as user ID:" && id
    - uv run ansible-builder build --tag $IMAGE_TAG --container-runtime podman
    - echo "$CI_REGISTRY_PASSWORD" | podman login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin
    - podman push $IMAGE_TAG

Actual behavior

Fails with the described Error from above.

Expected behavior

Builds and deploys the ee image to registry

Relevant logs and/or screenshots

job log

Running with gitlab-runner 18.2.1 (cc489270)
  on gitlab-runner-standalone RvNRv7UeL, system ID: s_df739cf4a37e
Resolving secrets
Preparing the "docker" executor 00:02
Using Docker executor with image quay.io/podman/stable:latest ...
Using effective pull policy of [always] for container quay.io/podman/stable:latest
Pulling docker image quay.io/podman/stable:latest ...
Using docker image sha256:58e7dcd4e2e8d0a25a32aeae35b474473f47e11cc6c2b212d2eb2a10047b3c78 for quay.io/podman/stable:latest with digest quay.io/podman/stable@sha256:20873d52795900323d1509d873eacb02b3cb686e8cd725c1cfe4861a847dd7c4 ...
Preparing environment 00:01
Using effective pull policy of [always] for container sha256:f87e5381a14cdcd3aa33d9cb65561dd8d8768740d971f35e6da2ff1914104f7c
Running on runner-rvnrv7uel-project-1284-concurrent-0 via gitlab-runner-standalone...
Getting source from Git repository 00:02
Gitaly correlation ID: 01KAX9SP63017H29JKKT10KDXY
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/fb10-it/ansible/gitlab-fb10/.git/
Created fresh repository.
Checking out 80529a1f as detached HEAD (ref is migrate-to-awx)...
Removing .venv/
Removing ee/context/
Removing ee/requirements.txt
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:26
Using effective pull policy of [always] for container quay.io/podman/stable:latest
Using docker image sha256:58e7dcd4e2e8d0a25a32aeae35b474473f47e11cc6c2b212d2eb2a10047b3c78 for quay.io/podman/stable:latest with digest quay.io/podman/stable@sha256:20873d52795900323d1509d873eacb02b3cb686e8cd725c1cfe4861a847dd7c4 ...
$ dnf install -y curl python3 python3-pip
Updating and loading repositories:
 Fedora 43 openh264 (From Cisco) - x86_ 100% |   7.0 KiB/s |   5.8 KiB |  00m01s
 Fedora 43 - x86_64 - Updates           100% |   9.7 MiB/s |  12.8 MiB |  00m01s
 Fedora 43 - x86_64                     100% |   6.4 MiB/s |  35.4 MiB |  00m06s
Repositories loaded.
Package "curl-8.15.0-3.fc43.x86_64" is already installed.
Package                     Arch   Version        Repository      Size
Installing:
 python3                    x86_64 3.14.0-2.fc43  updates     28.9 KiB
 python3-pip                noarch 25.1.1-18.fc43 fedora      12.5 MiB
Installing dependencies:
 mpdecimal                  x86_64 4.0.1-2.fc43   fedora     217.2 KiB
 python-pip-wheel           noarch 25.1.1-18.fc43 fedora       1.2 MiB
 python3-libs               x86_64 3.14.0-2.fc43  updates     43.0 MiB
Installing weak dependencies:
 python-unversioned-command noarch 3.14.0-2.fc43  updates     23.0   B
Transaction Summary:
 Installing:         6 packages
Total size of inbound packages is 14 MiB. Need to download 14 MiB.
After this operation, 57 MiB extra will be used (install 57 MiB, remove 0 B).
[1/6] python3-0:3.14.0-2.fc43.x86_64    100% |  29.8 KiB/s |  27.7 KiB |  00m01s
[2/6] mpdecimal-0:4.0.1-2.fc43.x86_64   100% | 776.9 KiB/s |  97.1 KiB |  00m00s
[3/6] python3-libs-0:3.14.0-2.fc43.x86_ 100% |   7.0 MiB/s |   9.8 MiB |  00m01s
[4/6] python-unversioned-command-0:3.14 100% | 278.4 KiB/s |   9.7 KiB |  00m00s
[5/6] python-pip-wheel-0:25.1.1-18.fc43 100% |   2.1 MiB/s |   1.2 MiB |  00m01s
[6/6] python3-pip-0:25.1.1-18.fc43.noar 100% |   1.6 MiB/s |   2.8 MiB |  00m02s
--------------------------------------------------------------------------------
[6/6] Total                             100% |   7.7 MiB/s |  14.0 MiB |  00m02s
Running transaction
[1/8] Verify package files              100% | 113.0   B/s |   6.0   B |  00m00s
[2/8] Prepare transaction               100% |  84.0   B/s |   6.0   B |  00m00s
[3/8] Installing python-pip-wheel-0:25. 100% |  29.6 MiB/s |   1.2 MiB |  00m00s
[4/8] Installing mpdecimal-0:4.0.1-2.fc 100% |  10.2 MiB/s | 218.8 KiB |  00m00s
[5/8] Installing python3-libs-0:3.14.0- 100% |  28.3 MiB/s |  43.3 MiB |  00m02s
[6/8] Installing python3-0:3.14.0-2.fc4 100% | 340.3 KiB/s |  30.6 KiB |  00m00s
[7/8] Installing python3-pip-0:25.1.1-1 100% |  38.7 MiB/s |  12.8 MiB |  00m00s
[8/8] Installing python-unversioned-com 100% | 340.0   B/s | 424.0   B |  00m01s
Complete!
$ curl -LsSf https://astral.sh/uv/install.sh | sh
downloading uv 0.9.11 x86_64-unknown-linux-gnu
no checksums to verify
installing to /root/.local/bin
  uv
  uvx
everything's installed!
To add $HOME/.local/bin to your PATH, either restart your shell or run:
    source $HOME/.local/bin/env (sh, bash, zsh)
    source $HOME/.local/bin/env.fish (fish)
$ source ~/.local/bin/env
$ uv --version
uv 0.9.11
$ uv pip compile pyproject.toml --group ee -o ee/requirements.txt
Resolved 14 packages in 155ms
# This file was autogenerated by uv via the following command:
#    uv pip compile pyproject.toml --group ee -o ee/requirements.txt
ansible-core==2.20.0
    # via gitlab-fb10 (pyproject.toml:ee)
ansible-runner==2.4.2
    # via gitlab-fb10 (pyproject.toml:ee)
cffi==2.0.0
    # via cryptography
cryptography==46.0.3
    # via ansible-core
jinja2==3.1.6
    # via ansible-core
lockfile==0.12.2
    # via python-daemon
markupsafe==3.0.3
    # via jinja2
packaging==25.0
    # via
    #   ansible-core
    #   ansible-runner
pexpect==4.9.0
    # via ansible-runner
ptyprocess==0.7.0
    # via pexpect
pycparser==2.23
    # via cffi
python-daemon==3.1.2
    # via ansible-runner
pyyaml==6.0.3
    # via
    #   ansible-core
    #   ansible-runner
resolvelib==1.2.1
    # via ansible-core
$ cd ee
$ echo "Running as user ID:" && id
Running as user ID:
uid=0(root) gid=0(root) groups=0(root)
$ uv run ansible-builder build --tag $IMAGE_TAG --container-runtime podman
Using CPython 3.14.0 interpreter at: /usr/bin/python3
Creating virtual environment at: /builds/fb10-it/ansible/gitlab-fb10/.venv
Downloading pygments (1.2MiB)
Downloading cryptography (4.3MiB)
Downloading setuptools (1.1MiB)
Downloading virtualenv (5.7MiB)
Downloading black (1.6MiB)
Downloading ansible-core (2.3MiB)
 Downloaded virtualenv
 Downloaded black
 Downloaded cryptography
 Downloaded pygments
 Downloaded setuptools
 Downloaded ansible-core
Installed 74 packages in 356ms
Running command:
  podman build -f context/Containerfile -t gitlab.fb10.fh-dortmund.de:5005/fb10-it/ansible/gitlab-fb10/gitlab-ee:latest context
cannot clone: Operation not permitted
Error: cannot re-exec process
An error occurred (rc=125), see output line(s) above for details.
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

Environment description

docker info

Client: Docker Engine - Community Version: 29.0.4 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.30.0 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.40.3 Path: /usr/libexec/docker/cli-plugins/docker-compose

Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 3 Server Version: 29.0.4 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog CDI spec directories: /etc/cdi /var/run/cdi Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: fcd43222d6b07379a4be9786bda52438f0dd16a1 runc version: v1.3.3-0-gd842d771 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.8.0-88-generic Operating System: Ubuntu 24.04.3 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 3.823GiB Name: gitlab-runner-docker ID: *** Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: ::1/128 127.0.0.0/8 Live Restore Enabled: false Firewall Backend: iptables

config.toml contents

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "docker-runner"
  url = "https://gitlab.fb10.fh-dortmund.de"
  request_concurrency = 4
  id = 35
  token = "***"
  token_obtained_at = 2024-11-22T16:57:52Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ['/cache']
    shm_size = 0
    network_mtu = 0

Used GitLab Runner version

Before:

Version:      17.11.0
Git revision: 0f67ff19
Git branch:   17-11-stable
GO version:   go1.23.6 X:cacheprog
Built:        2025-04-14T10:18:18Z
OS/Arch:      linux/amd64

After:

Version:      18.6.1
Git revision: b5e9c6d0
Git branch:   18-6-stable
GO version:   go1.24.6 X:cacheprog
Built:        2025-11-21T06:34:41Z
OS/Arch:      linux/amd64