Clear-docker-cache: fall back to podman when docker is absent (!6755) · Merge requests · GitLab.org / gitlab-runner

What does this MR do?

clear-docker-cache: fall back to podman when docker is absent

Detect the container engine at startup: prefer docker, fall back to podman, allow override via $CONTAINER_ENGINE. Substitute the binary into every command invocation in the script. Skip the Docker API-version gate when running under Podman (Podman uses its own API versioning and supports the modern prune commands the script issues).

For rootless Podman, the cron entry must still be run as the user that owns the Podman storage; that constraint is inherent to rootless storage and is now noted in the docs.

Closes #36934 (closed)

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

In depth detailed explanation of trade offs and design decision making:

clear-docker-cache: fall back to podman when docker is absent
Closes #36934 (closed).

The bug:

The bundled clear-docker-cache script (which users run from cron to prune runner-managed containers, networks, images, and volumes) checks for a docker binary at startup. If docker isn't on PATH, it prints INFO: Docker installation not found, skipping clear-docker-cache and exits 0.
For installs that use Podman as the docker-API-compatible engine — a setup the runner has officially supported since 15.3 — the script therefore does nothing. Podman storage grows indefinitely; the user has no cleanup path other than docker system prune aliases, hand-rolled wrappers, or running the engine commands themselves on cron. The issue has been open since 2023 with no documented workaround in-tree.

The fix:

Detect the container engine at script startup:

Preferdocker. If it's onPATH, setCONTAINER_ENGINE=dockerand behave identically to the previous script.

Ifdockeris not onPATH, look forpodman. If found, setCONTAINER_ENGINE=podmanand proceed.

If neither is found, print aWARNING:and exit 0 (preserving the no-spam behavior for hosts where no engine is expected).

Then substitute "$CONTAINER_ENGINE" for the hard-coded docker in every command invocation in the script, and gate the docker-specific API-version check (< 1.25) on CONTAINER_ENGINE = docker so that podman doesn't fail it.
Net diff: 2 files, ~30 lines changed.

Why this approach:

This was a deliberately scoped choice. The alternative considered was implementing a proper gitlab-runner clear-docker-cache Go subcommand that reads config.toml, discovers every docker/podman host referenced by a [[runners]] block, and prunes each. That approach has real upsides — zero user-config burden, native config parsing, testable in Go — but it is also a ~500-line change, harder to backport to stable branches, and introduces a new subcommand surface to maintain. It belongs in a feature MR, not in the patch that fixes a 2-year-old bug.

The bash fallback gives users the unblock now and keeps the door open for the Go version later (the legacy script can become a shim to a subcommand in a future MR — see the fix/issue-36934-clear-docker-cache-podman branch where that was prototyped).

Design decisions:

Preference order: docker first, podman fallback

Docker is preferred because that's the script's historical contract — every user who has docker installed sees identical behavior. Falling back to podman only when docker isn't on PATH means the change is a strict superset of the previous behavior for any setup where docker actually works. There's no risk of accidentally switching a working docker user onto podman.

INFO: messages narrate the engine-detection flow

A user whose docker binary is installed off-PATH (e.g. /usr/local/bin/docker while cron's PATH=/usr/bin:/bin) and who also has podman installed will silently start pruning podman objects instead of doing nothing. To make that visible in cron mail:

INFO: docker not found, checking for podman... INFO: podman found, using podman

The messages narrate the actual decision flow top-to-bottom so the log reads like the code does. Combined with the default filter (label=com.gitlab.gitlab-runner.managed=true), the blast radius of an unexpected fallback is bounded to "runner-managed podman objects" — which is the right scope by construction.
WARNING: (not ERROR:) for the "neither found" case

Exiting 0 keeps cron from emitting mail every run on hosts that legitimately don't have either engine — consistent with the previous script's design choice. The WARNING: prefix bumps the log level so anyone tailing cron output can see something unusual happened, without escalating to a job failure.
No CONTAINER_ENGINE environment-variable override

An earlier draft made the engine overridable via env var. Dropped: no consumer in-tree needs it, and the only realistic use case ("I have both engines installed and want to force podman") is solved by uninstalling docker — which is what someone in that state probably wants anyway. Keeping the API surface to "auto-detect" is one fewer thing to document.
No rename of the script

The file is still called clear-docker-cache, even though it now handles podman. Renaming would mean either (a) adding a new file at clear-container-cache and turning the old name into a shim, or (b) doing a hard rename. Both options require touching packaging manifests, install scripts in deb/rpm/chef/ansible/puppet across the ecosystem, and the docs in multiple locales. None of that belongs in a bug-fix MR. The name is a wart but it's not a bug, and renaming can land in a focused cleanup MR later.

Rootless podman caveat: docs only, not enforced

Rootless podman stores containers and images under $HOME/.local/share/containers/storage, per-user, not in a shared system location. If cron runs as root against a rootless setup, root sees its own (empty) podman storage, not the gitlab-runner user's storage. The script can detect podman's binary but it can't fix this — it's the inherent contract of rootless containers.

The docs paragraph explicitly explains why "rootless" is the qualifier (storage location is the reason), and says cron must run as the user who owns the storage. Rootful podman doesn't have this constraint and isn't called out; if you're rootful, you're effectively docker-equivalent for this purpose.
What this MR explicitly does not do

Does not read config.toml. No new dependencies. The script behaves the same way regardless of how[[runners]]is configured. A user running multiple runners against different sockets is out of scope for this fix.

**Does not add tests.**The script is short, the change is mechanical, and the test infrastructure required to mock docker/podman binaries reliably across distros would be larger than the fix. Validated locally with a mock-binary harness that exercises all five matrix cells (docker only, podman only, both, neither, podman with prune-volumes); details in the MR thread.

**Does not change the script's CLI.**Same positional argument (prune/prune-volumes/space/help), same default action (prune-volumes), sameFILTER_FLAGenv var, same exit codes.

**Does not change the docker code path.**Anyone running docker sees byte-identical behavior aside from the variable-substitution layer; theawkcomparisons land on the same branches and issue the same commands.

Backport considerations:

The change is bash-only and doesn't touch any Go code. Stable branches that ship the same clear-docker-cache script should backport cleanly with a manual git cherry-pick. The diff is small enough that conflict resolution against older versions of the script (if any thresholds or messages changed historically) is straightforward.
Testing performed

Local verification with mock docker and podman shell scripts on PATH, exercising:

docker present—CONTAINER_ENGINE=dockerset, commands invoked unchanged. No INFO/WARNING output.

podman only— INFO messages fire in order,podman system prune -af --filter ...(andvolume prunefor the defaultprune-volumesaction) invoked with the modern flags.

neither present— WARNING fires, exit 0.

both present— docker wins, podman never invoked.

prune vs prune-volumes actions under podman— onlyprune-volumescallsvolume prune;prunedoes not.

No regression in any existing docker scenario; new podman scenario does what the issue asks for.

Why was this MR needed?

What's the best way to test this MR?

Adding a runbook using lima for testing purposes. Tested podman locally on my Lima Ubuntu container:

Reproduction against real podman

Ran end-to-end in a fresh Ubuntu 24.04 VM (Lima on macOS, but any Linux works) with podman 4.9.3 installed.

Setup (one-time)

`brew install lima limactl start --name=podman-test template:ubuntu-24.04 limactl shell podman-test -- sudo apt-get update -qq limactl shell podman-test -- sudo apt-get install -y -qq podman uidmap

SCRIPT=/Users/lachiegrant/Git/gitlab-runner/packaging/root/usr/share/gitlab-runner/clear-docker-cache `

1. Script detects podman

limactl shell podman-test -- bash "$SCRIPT" help

Output:

`INFO: docker not found, checking for podman... INFO: podman found, using podman

Usage: ... prune-volumes|prune|space|help ... `

2. `prune` only removes runner-managed containers

limactl shell podman-test -- bash -c ' podman run -d --label com.gitlab.gitlab-runner.managed=true alpine true podman run -d alpine true sleep 1 podman ps -a --format " {{.ID}} {{.Labels}}" '

Output:

b5bc9aba8cad map[com.gitlab.gitlab-runner.managed:true] d8ea33783486 map[]

limactl shell podman-test -- bash "$SCRIPT" prune limactl shell podman-test -- podman ps -a --format " {{.ID}} {{.Labels}}"

Output:

`Deleted Containers b5bc9aba8cad33055ef3ab3b5802e253adb3834fd0d3f622e7e6d43173939577 Total reclaimed space: 11.08kB

d8ea33783486 map[] `

The labelled container is gone; the unlabelled one is preserved — the default filter label=com.gitlab.gitlab-runner.managed=true is doing its job against real podman.

3. Rootless caveat — cron must run as the storage owner

limactl shell podman-test -- sudo useradd -m gitlab-runner limactl shell podman-test -- sudo loginctl enable-linger gitlab-runner limactl shell podman-test -- sudo -iu gitlab-runner bash -c ' podman run -d --label com.gitlab.gitlab-runner.managed=true alpine true ' limactl shell podman-test -- sudo -iu gitlab-runner podman ps -a --format " {{.ID}} {{.Labels}}"

Output (gitlab-runner user has one labelled container in their storage):

cd5263870701 map[com.gitlab.gitlab-runner.managed:true]

Run the script as root (simulating cron-as-root):

limactl shell podman-test -- sudo bash "$SCRIPT" prune limactl shell podman-test -- sudo -iu gitlab-runner podman ps -a --format " {{.ID}} {{.Labels}}"

Output:

`Total reclaimed space: 0B

cd5263870701 map[com.gitlab.gitlab-runner.managed:true] `

Root prune reclaimed 0B — it cleaned root's empty podman storage, not gitlab-runner's. The container survives. This is exactly what the docs caveat warns about.

Run as the right user:

limactl shell podman-test -- sudo -iu gitlab-runner bash "$SCRIPT" prune limactl shell podman-test -- sudo -iu gitlab-runner podman ps -a --format " {{.ID}} {{.Labels}}"

Output:

`Deleted Containers cd5263870701807e956d5a8cc4f7c9fbb5ed9b9cfadbf435c17c6a93968e57ca Total reclaimed space: 11.13kB

(empty) `

Summary

✅ Script finds real podman (bug fix works)
✅ Modern system prune -af --filter syntax accepted by podman natively
✅ Default label filter targets only runner-managed objects
✅ Rootless-storage caveat reproduced — root cron'd against rootless user = 0B reclaimed; same script as the storage owner = works

Teardown

limactl stop podman-test limactl delete podman-test

Bash script for testing against a mock docker / podman path, which just prints out the commands (doesn't execute agains the real podman or docker binary).

#!/usr/bin/env bash
# Local test harness for packaging/root/usr/share/gitlab-runner/clear-docker-cache.
#
# Verifies engine detection and command routing under every relevant combo of
# docker/podman/none on PATH, plus every docker version threshold the script
# gates on. Uses mock `docker` and `podman` shell scripts placed on PATH —
# neither engine needs to be installed.
#
# Usage: bash scripts/test-clear-docker-cache.sh
# Exit:  0 if all tests pass, 1 otherwise.

set -uo pipefail

SCRIPT="$(cd "$(dirname "$0")/.." && pwd)/packaging/root/usr/share/gitlab-runner/clear-docker-cache"
TMPDIR="$(mktemp -d)"
trap 'rm -rf "$TMPDIR"' EXIT

PASS=0
FAIL=0
RC=0

# Create $TMPDIR/bin with a fresh slate (no engines installed).
setup_clean_bin() {
  rm -rf "$TMPDIR/bin"
  mkdir -p "$TMPDIR/bin"
}

# Drop a mock engine binary into $TMPDIR/bin.
#   $1 = engine name (docker | podman)
#   $2 = client version reported (e.g. 24.0.7)
#   $3 = api version reported    (e.g. 1.43)
make_mock() {
  local name="$1" cver="$2" apiver="$3"
  cat > "$TMPDIR/bin/$name" <<EOF
#!/usr/bin/env bash
echo "$name: \$*" >> "$TMPDIR/calls.log"
case "\$1" in
  version)
    case "\$3" in
      *Client.APIVersion*) echo "$apiver" ;;
      *Client.Version*) echo "$cver" ;;
    esac ;;
  system) case "\$2" in df) echo "(df)";; prune) echo "(system prune)";; esac ;;
  volume) case "\$2" in prune) echo "(volume prune)";; esac ;;
  ps) echo "" ;;  # no containers
esac
EOF
  chmod +x "$TMPDIR/bin/$name"
}

# Run the script under test, capture exit + stdout/stderr + recorded calls.
#   $1 = action (prune | prune-volumes | space | help)
run_with_bin() {
  local action="${1:-prune-volumes}"
  : > "$TMPDIR/calls.log"
  : > "$TMPDIR/stdout.log"
  : > "$TMPDIR/stderr.log"
  PATH="$TMPDIR/bin:/usr/bin:/bin" \
    bash "$SCRIPT" "$action" > "$TMPDIR/stdout.log" 2> "$TMPDIR/stderr.log"
  RC=$?
}

test_case() {
  echo ""
  echo "=== $1 ==="
}

assert_rc() {
  if [ "$RC" = "$1" ]; then
    echo "  PASS  exit code = $1"
    PASS=$((PASS+1))
  else
    echo "  FAIL  expected exit $1, got $RC"
    FAIL=$((FAIL+1))
  fi
}

# Assert that the mocks recorded a call matching the literal substring.
assert_called() {
  if grep -qF "$1" "$TMPDIR/calls.log"; then
    echo "  PASS  called: $1"
    PASS=$((PASS+1))
  else
    echo "  FAIL  expected call NOT found: $1"
    echo "        actual calls:"
    sed 's/^/          /' "$TMPDIR/calls.log"
    FAIL=$((FAIL+1))
  fi
}

assert_not_called() {
  if ! grep -qF "$1" "$TMPDIR/calls.log"; then
    echo "  PASS  not called: $1"
    PASS=$((PASS+1))
  else
    echo "  FAIL  unexpected call FOUND: $1"
    FAIL=$((FAIL+1))
  fi
}

assert_output_contains() {
  if grep -qF "$1" "$TMPDIR/stdout.log" || grep -qF "$1" "$TMPDIR/stderr.log"; then
    echo "  PASS  output contains: $1"
    PASS=$((PASS+1))
  else
    echo "  FAIL  output missing:  $1"
    echo "        stdout:"
    sed 's/^/          /' "$TMPDIR/stdout.log"
    FAIL=$((FAIL+1))
  fi
}

###############################################################################
# Engine detection
###############################################################################

test_case "docker only on PATH, prune-volumes -> modern path, no INFO/WARN"
setup_clean_bin
make_mock docker 24.0.7 1.43
run_with_bin prune-volumes
assert_rc 0
assert_called "docker: system prune -af --filter label=com.gitlab.gitlab-runner.managed=true"
assert_called "docker: volume prune -af --filter label=com.gitlab.gitlab-runner.managed=true"

test_case "podman only on PATH, prune-volumes -> falls back, modern path"
setup_clean_bin
make_mock podman 5.0.0 4.0
run_with_bin prune-volumes
assert_rc 0
assert_output_contains "INFO: docker not found, checking for podman"
assert_output_contains "INFO: podman found, using podman"
assert_called "podman: system prune -af --filter"
assert_called "podman: volume prune -af --filter"

test_case "both engines installed -> docker wins, podman never invoked"
setup_clean_bin
make_mock docker 24.0.7 1.43
make_mock podman 5.0.0 4.0
run_with_bin prune
assert_rc 0
assert_called "docker: system prune -af --filter"
assert_not_called "podman:"

test_case "neither engine installed -> WARNING, exit 0 (no cron mail)"
setup_clean_bin
run_with_bin prune
assert_rc 0
assert_output_contains "WARNING: neither docker nor podman found"

###############################################################################
# Reviewer-flagged concern: podman MUST reach the else branch in prune-volumes
###############################################################################

test_case "REVIEWER POINT 1: podman + prune-volumes does NOT fall through"
setup_clean_bin
make_mock podman 5.0.0 4.0
run_with_bin prune-volumes
assert_rc 0
# If the four [ docker ] && ... branches gated the else too, both of these would
# fail and the script would silently exit 0 without pruning anything.
assert_called "podman: system prune -af --filter"
assert_called "podman: volume prune -af --filter"

###############################################################################
# Docker version-gated branches
###############################################################################

test_case "docker 17.05.0 (< 17.06.1), prune -> legacy 'docker rm' path"
setup_clean_bin
make_mock docker 17.05.0 1.29
run_with_bin prune
assert_rc 0
assert_called "docker: ps -a -q"
# The mock returns no containers, so 'rm' isn't called; key is 'system prune' isn't either.
assert_not_called "docker: system prune"

test_case "docker 17.03.0 (< 17.04.0), prune-volumes -> 'docker rm -v' path"
setup_clean_bin
make_mock docker 17.03.0 1.27
run_with_bin prune-volumes
assert_rc 0
assert_called "docker: ps -a -q"
assert_not_called "docker: system prune"
assert_not_called "docker: volume prune"

test_case "docker 17.04.5 (17.04 <= v < 17.05), prune-volumes -> system prune only"
setup_clean_bin
make_mock docker 17.04.5 1.28
run_with_bin prune-volumes
assert_rc 0
assert_called "docker: system prune -af --filter"
assert_not_called "docker: volume prune"

test_case "docker 20.10.0 (17.05 <= v < 23.0), prune-volumes -> volume prune -f"
setup_clean_bin
make_mock docker 20.10.0 1.41
run_with_bin prune-volumes
assert_rc 0
assert_called "docker: system prune -af --filter"
assert_called "docker: volume prune -f --filter"
assert_not_called "docker: volume prune -af"

test_case "docker 23.0.0 (>= 23.0), prune-volumes -> volume prune -af"
setup_clean_bin
make_mock docker 23.0.0 1.42
run_with_bin prune-volumes
assert_rc 0
assert_called "docker: volume prune -af --filter"

test_case "docker API version < 1.25 -> error and exit 1"
setup_clean_bin
make_mock docker 1.13.0 1.24
run_with_bin prune
assert_rc 1
assert_output_contains "ERROR: Your current API version is lower than 1.25"

test_case "podman with low 'API version' string -> does NOT trip the docker API gate"
setup_clean_bin
# Set an APIVersion that would lexically sort below 1.25 if compared, e.g. "1.0".
# The [ docker ] guard must short-circuit so podman is unaffected.
make_mock podman 5.0.0 1.0
run_with_bin prune
assert_rc 0
assert_called "podman: system prune -af --filter"

###############################################################################
# Other actions
###############################################################################

test_case "podman, space -> system df"
setup_clean_bin
make_mock podman 5.0.0 4.0
run_with_bin space
assert_rc 0
assert_called "podman: system df"

test_case "docker, space -> system df"
setup_clean_bin
make_mock docker 24.0.7 1.43
run_with_bin space
assert_rc 0
assert_called "docker: system df"

test_case "docker, help -> usage block, exit 1"
setup_clean_bin
make_mock docker 24.0.7 1.43
run_with_bin help
assert_rc 1
assert_output_contains "Usage: "
assert_output_contains "prune-volumes"

###############################################################################
# Summary
###############################################################################

echo ""
echo "================================="
echo "  PASS: $PASS"
echo "  FAIL: $FAIL"
echo "================================="

if [ "$FAIL" -gt 0 ]; then
  exit 1
fi
exit 0

What are the relevant issue numbers?

Edited May 18, 2026 by Lachlan Grant

Clear-docker-cache: fall back to podman when docker is absent

What does this MR do?

Why was this MR needed?

What's the best way to test this MR?

Reproduction against real podman

Setup (one-time)

1. Script detects podman

2. prune only removes runner-managed containers

3. Rootless caveat — cron must run as the storage owner

Summary

Teardown

What are the relevant issue numbers?

Merge request reports

2. `prune` only removes runner-managed containers