Executing podman inside docker runner
## Summary
I had some issue with space on one my runners and used this circumstance to also upgrade the runner.
The runner is configured as linux runner with docker executor.
After Upgraded from manual installation to debian poackage.
Version: 17.11.0 -> 18.6.1
After the upgrade the job I was running fails with:
```bash
Running command:
podman build -f context/Containerfile -t gitlab.fb10.fh-dortmund.de:5005/fb10-it/ansible/gitlab-fb10/gitlab-ee:latest context
cannot clone: Operation not permitted
Error: cannot re-exec process
```
I try to build an ansible execution environment which uses podman to build and push the image to registry.
## Steps to reproduce
I provide the masked runner config and gitlab-ci.yml for this job.
Additional I provide a simple exection_environment.yml file which is used to build the ee.
In previous runs with 17.11.0 the runner executed further than the failed command.
<details>
<summary> ee/execution-environment.yml </summary>
```yml
version: 3
images:
base_image:
name: registry.fedoraproject.org/fedora:42
dependencies:
python_interpreter:
package_system: python3
ansible_core:
package_pip: ansible-core
ansible_runner:
package_pip: ansible-runner
system:
- openssh-clients
- sshpass
```
</details>
<!--
What do you need to do to reproduce the bug? Please include job definitions or git repository structure if relevant
-->
<details>
<summary> .gitlab-ci.yml </summary>
```yml
variables:
UV_VERSION: "0.5"
PYTHON_VERSION: "3.12"
BASE_LAYER: bookworm-slim
UV_LINK_MODE: copy
IMAGE_TAG: $CI_REGISTRY_IMAGE/gitlab-ee:latest
stages:
- build
build-and-deploy-ee:
image: quay.io/podman/stable:latest
stage: build
variables:
STORAGE_DRIVER: vfs
before_script:
- dnf install -y curl python3 python3-pip
- curl -LsSf https://astral.sh/uv/install.sh | sh
- source ~/.local/bin/env
- uv --version
script:
- uv pip compile pyproject.toml --group ee -o ee/requirements.txt
- cd ee
- echo "Running as user ID:" && id
- uv run ansible-builder build --tag $IMAGE_TAG --container-runtime podman
- echo "$CI_REGISTRY_PASSWORD" | podman login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin
- podman push $IMAGE_TAG
```
</details>
## Actual behavior
Fails with the described Error from above.
## Expected behavior
Builds and deploys the ee image to registry
## Relevant logs and/or screenshots
<!--
Paste the job logs inside of the code blocks (```) below so it would be
easier to read.
-->
<details>
<summary> job log </summary>
```sh
Running with gitlab-runner 18.2.1 (cc489270)
on gitlab-runner-standalone RvNRv7UeL, system ID: s_df739cf4a37e
Resolving secrets
Preparing the "docker" executor 00:02
Using Docker executor with image quay.io/podman/stable:latest ...
Using effective pull policy of [always] for container quay.io/podman/stable:latest
Pulling docker image quay.io/podman/stable:latest ...
Using docker image sha256:58e7dcd4e2e8d0a25a32aeae35b474473f47e11cc6c2b212d2eb2a10047b3c78 for quay.io/podman/stable:latest with digest quay.io/podman/stable@sha256:20873d52795900323d1509d873eacb02b3cb686e8cd725c1cfe4861a847dd7c4 ...
Preparing environment 00:01
Using effective pull policy of [always] for container sha256:f87e5381a14cdcd3aa33d9cb65561dd8d8768740d971f35e6da2ff1914104f7c
Running on runner-rvnrv7uel-project-1284-concurrent-0 via gitlab-runner-standalone...
Getting source from Git repository 00:02
Gitaly correlation ID: 01KAX9SP63017H29JKKT10KDXY
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/fb10-it/ansible/gitlab-fb10/.git/
Created fresh repository.
Checking out 80529a1f as detached HEAD (ref is migrate-to-awx)...
Removing .venv/
Removing ee/context/
Removing ee/requirements.txt
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:26
Using effective pull policy of [always] for container quay.io/podman/stable:latest
Using docker image sha256:58e7dcd4e2e8d0a25a32aeae35b474473f47e11cc6c2b212d2eb2a10047b3c78 for quay.io/podman/stable:latest with digest quay.io/podman/stable@sha256:20873d52795900323d1509d873eacb02b3cb686e8cd725c1cfe4861a847dd7c4 ...
$ dnf install -y curl python3 python3-pip
Updating and loading repositories:
Fedora 43 openh264 (From Cisco) - x86_ 100% | 7.0 KiB/s | 5.8 KiB | 00m01s
Fedora 43 - x86_64 - Updates 100% | 9.7 MiB/s | 12.8 MiB | 00m01s
Fedora 43 - x86_64 100% | 6.4 MiB/s | 35.4 MiB | 00m06s
Repositories loaded.
Package "curl-8.15.0-3.fc43.x86_64" is already installed.
Package Arch Version Repository Size
Installing:
python3 x86_64 3.14.0-2.fc43 updates 28.9 KiB
python3-pip noarch 25.1.1-18.fc43 fedora 12.5 MiB
Installing dependencies:
mpdecimal x86_64 4.0.1-2.fc43 fedora 217.2 KiB
python-pip-wheel noarch 25.1.1-18.fc43 fedora 1.2 MiB
python3-libs x86_64 3.14.0-2.fc43 updates 43.0 MiB
Installing weak dependencies:
python-unversioned-command noarch 3.14.0-2.fc43 updates 23.0 B
Transaction Summary:
Installing: 6 packages
Total size of inbound packages is 14 MiB. Need to download 14 MiB.
After this operation, 57 MiB extra will be used (install 57 MiB, remove 0 B).
[1/6] python3-0:3.14.0-2.fc43.x86_64 100% | 29.8 KiB/s | 27.7 KiB | 00m01s
[2/6] mpdecimal-0:4.0.1-2.fc43.x86_64 100% | 776.9 KiB/s | 97.1 KiB | 00m00s
[3/6] python3-libs-0:3.14.0-2.fc43.x86_ 100% | 7.0 MiB/s | 9.8 MiB | 00m01s
[4/6] python-unversioned-command-0:3.14 100% | 278.4 KiB/s | 9.7 KiB | 00m00s
[5/6] python-pip-wheel-0:25.1.1-18.fc43 100% | 2.1 MiB/s | 1.2 MiB | 00m01s
[6/6] python3-pip-0:25.1.1-18.fc43.noar 100% | 1.6 MiB/s | 2.8 MiB | 00m02s
--------------------------------------------------------------------------------
[6/6] Total 100% | 7.7 MiB/s | 14.0 MiB | 00m02s
Running transaction
[1/8] Verify package files 100% | 113.0 B/s | 6.0 B | 00m00s
[2/8] Prepare transaction 100% | 84.0 B/s | 6.0 B | 00m00s
[3/8] Installing python-pip-wheel-0:25. 100% | 29.6 MiB/s | 1.2 MiB | 00m00s
[4/8] Installing mpdecimal-0:4.0.1-2.fc 100% | 10.2 MiB/s | 218.8 KiB | 00m00s
[5/8] Installing python3-libs-0:3.14.0- 100% | 28.3 MiB/s | 43.3 MiB | 00m02s
[6/8] Installing python3-0:3.14.0-2.fc4 100% | 340.3 KiB/s | 30.6 KiB | 00m00s
[7/8] Installing python3-pip-0:25.1.1-1 100% | 38.7 MiB/s | 12.8 MiB | 00m00s
[8/8] Installing python-unversioned-com 100% | 340.0 B/s | 424.0 B | 00m01s
Complete!
$ curl -LsSf https://astral.sh/uv/install.sh | sh
downloading uv 0.9.11 x86_64-unknown-linux-gnu
no checksums to verify
installing to /root/.local/bin
uv
uvx
everything's installed!
To add $HOME/.local/bin to your PATH, either restart your shell or run:
source $HOME/.local/bin/env (sh, bash, zsh)
source $HOME/.local/bin/env.fish (fish)
$ source ~/.local/bin/env
$ uv --version
uv 0.9.11
$ uv pip compile pyproject.toml --group ee -o ee/requirements.txt
Resolved 14 packages in 155ms
# This file was autogenerated by uv via the following command:
# uv pip compile pyproject.toml --group ee -o ee/requirements.txt
ansible-core==2.20.0
# via gitlab-fb10 (pyproject.toml:ee)
ansible-runner==2.4.2
# via gitlab-fb10 (pyproject.toml:ee)
cffi==2.0.0
# via cryptography
cryptography==46.0.3
# via ansible-core
jinja2==3.1.6
# via ansible-core
lockfile==0.12.2
# via python-daemon
markupsafe==3.0.3
# via jinja2
packaging==25.0
# via
# ansible-core
# ansible-runner
pexpect==4.9.0
# via ansible-runner
ptyprocess==0.7.0
# via pexpect
pycparser==2.23
# via cffi
python-daemon==3.1.2
# via ansible-runner
pyyaml==6.0.3
# via
# ansible-core
# ansible-runner
resolvelib==1.2.1
# via ansible-core
$ cd ee
$ echo "Running as user ID:" && id
Running as user ID:
uid=0(root) gid=0(root) groups=0(root)
$ uv run ansible-builder build --tag $IMAGE_TAG --container-runtime podman
Using CPython 3.14.0 interpreter at: /usr/bin/python3
Creating virtual environment at: /builds/fb10-it/ansible/gitlab-fb10/.venv
Downloading pygments (1.2MiB)
Downloading cryptography (4.3MiB)
Downloading setuptools (1.1MiB)
Downloading virtualenv (5.7MiB)
Downloading black (1.6MiB)
Downloading ansible-core (2.3MiB)
Downloaded virtualenv
Downloaded black
Downloaded cryptography
Downloaded pygments
Downloaded setuptools
Downloaded ansible-core
Installed 74 packages in 356ms
Running command:
podman build -f context/Containerfile -t gitlab.fb10.fh-dortmund.de:5005/fb10-it/ansible/gitlab-fb10/gitlab-ee:latest context
cannot clone: Operation not permitted
Error: cannot re-exec process
An error occurred (rc=125), see output line(s) above for details.
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1
```
</details>
## Environment description
<details>
<summary> docker info </summary>
Client: Docker Engine - Community
Version: 29.0.4
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.30.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.40.3
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 3
Server Version: 29.0.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: fcd43222d6b07379a4be9786bda52438f0dd16a1
runc version: v1.3.3-0-gd842d771
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-88-generic
Operating System: Ubuntu 24.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.823GiB
Name: gitlab-runner-docker
ID: ***
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Firewall Backend: iptables
</details>
<!--
Please add the contents of `config.toml` inside of the code blocks (```)
below, remember to remove any secret tokens!
-->
<details>
<summary> config.toml contents </summary>
```toml
concurrent = 1
check_interval = 0
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "docker-runner"
url = "https://gitlab.fb10.fh-dortmund.de"
request_concurrency = 4
id = 35
token = "***"
token_obtained_at = 2024-11-22T16:57:52Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ['/cache']
shm_size = 0
network_mtu = 0
```
</details>
### Used GitLab Runner version
Before:
```
Version: 17.11.0
Git revision: 0f67ff19
Git branch: 17-11-stable
GO version: go1.23.6 X:cacheprog
Built: 2025-04-14T10:18:18Z
OS/Arch: linux/amd64
```
After:
```
Version: 18.6.1
Git revision: b5e9c6d0
Git branch: 18-6-stable
GO version: go1.24.6 X:cacheprog
Built: 2025-11-21T06:34:41Z
OS/Arch: linux/amd64
```
issue