Reap leaked podman bridge interfaces on workers
Problem
netavark occasionally fails to delete a bridge during container teardown: the podman network config and IPAM lease are removed, but the kernel podmanN bridge survives as an orphan with no owning network. These accumulate on long-running workers and eventually collide with a freshly created network:
netavark: create bridge: Netlink error: File exists (os error 17)which fails container jobs with a spurious "permission denied" (exit 126).
For example, testing-farm-worker-4a04c9c8 had 22 such orphans (podman1–podman22, subnets 10.89.0.0/24–10.89.21.0/24) accumulated over 39 days of uptime.
Investigation
- Confirmed the orphans have no owning podman network (
/etc/containers/networksempty,podman network lsshows only the default) and no attached containers — pure leaks. - Ruled out
tmt: its create→network rmsequence is correct and runs cleanup in afinally. ~165 manual create/run/teardown cycles on the host — clean, concurrent, and interrupted (incl.SIGKILLmid-teardown) — leaked 0 bridges. The leak is a rarenetavarkteardown race, not atmtlogic bug. - The bridge is created lazily at first container attach and removed by
netavarkon detach; the orphan arises when that teardown silently fails to remove the bridge.
Change
Adds a podman-bridge-reaper systemd timer (hourly, monotonic) that deletes only provably-unused bridges — requiring all of:
- name is
podmanN(N >= 1; the defaultpodman0is never touched), - not the interface of any existing podman network, and
- no enslaved interface (no container attached).
Wired into tasks/podman.yaml, so both public and Red Hat workers pick it up on provision.
Validation
Deployed manually to testing-farm-worker-4a04c9c8 only (not fleet-wide yet):
- First run reaped all 22 orphans; orphan count now 0.
- Selective-reap test: with a live network + running container (
podman2) and a synthetic orphan (podman99) present, the reaper removed onlypodman99and left the live bridge and its container untouched. - Networking functional check (create → run → rm) passes;
podman0intact. - Timer enabled and firing hourly; was watched for a few hours.
Resolves TFT-4726
Assisted-by: Claude Code