Reap leaked podman bridge interfaces on workers

Problem

netavark occasionally fails to delete a bridge during container teardown: the podman network config and IPAM lease are removed, but the kernel podmanN bridge survives as an orphan with no owning network. These accumulate on long-running workers and eventually collide with a freshly created network:

netavark: create bridge: Netlink error: File exists (os error 17)

which fails container jobs with a spurious "permission denied" (exit 126).

For example, testing-farm-worker-4a04c9c8 had 22 such orphans (podman1podman22, subnets 10.89.0.0/2410.89.21.0/24) accumulated over 39 days of uptime.

Investigation

  • Confirmed the orphans have no owning podman network (/etc/containers/networks empty, podman network ls shows only the default) and no attached containers — pure leaks.
  • Ruled out tmt: its create→network rm sequence is correct and runs cleanup in a finally. ~165 manual create/run/teardown cycles on the host — clean, concurrent, and interrupted (incl. SIGKILL mid-teardown) — leaked 0 bridges. The leak is a rare netavark teardown race, not a tmt logic bug.
  • The bridge is created lazily at first container attach and removed by netavark on detach; the orphan arises when that teardown silently fails to remove the bridge.

Change

Adds a podman-bridge-reaper systemd timer (hourly, monotonic) that deletes only provably-unused bridges — requiring all of:

  • name is podmanN (N >= 1; the default podman0 is never touched),
  • not the interface of any existing podman network, and
  • no enslaved interface (no container attached).

Wired into tasks/podman.yaml, so both public and Red Hat workers pick it up on provision.

Validation

Deployed manually to testing-farm-worker-4a04c9c8 only (not fleet-wide yet):

  • First run reaped all 22 orphans; orphan count now 0.
  • Selective-reap test: with a live network + running container (podman2) and a synthetic orphan (podman99) present, the reaper removed only podman99 and left the live bridge and its container untouched.
  • Networking functional check (create → run → rm) passes; podman0 intact.
  • Timer enabled and firing hourly; was watched for a few hours.

Resolves TFT-4726

Assisted-by: Claude Code

Edited by Miroslav Vadkerti

Merge request reports

Loading