disconnectNetwork logs successful disconnects as WARNING since 19.2.0

Summary

Since upgrading to 19.2.0, every job ends with WARNING: Possibly zombie container ... is disconnected from network ... lines during cleanup. This message is emitted from the success branch of disconnectNetwork() — it reports a cleanup that worked, not a failure. Users read it as an error and open support tickets.

The behaviour change comes from commit 869d9e0e8 ("Fix disconnectNetwork container ID matching"), which made the container matching work for the first time. The log line itself is pre-existing, but until 19.2.0 it was effectively unreachable.

Steps to reproduce

  1. Run GitLab Runner 19.2.0 with the docker executor
  2. Run any job
  3. Observe the Cleaning up project directory and file based variables section

Actual behaviour

Cleaning up project directory and file based variables WARNING: Possibly zombie container runner-xxxxxxx-project-1-concurrent-0-predefined is disconnected from network bridge WARNING: Possibly zombie container runner-xxxxxxx-project-1-concurrent-0-build is disconnected from network bridge Job succeeded

Two warnings on every single job. No such messages on 19.1.1 with the same configuration.

Expected behaviour

A successful network disconnect should not be logged at WARNING level, and should not describe the container as a "possibly zombie" one when the operation completed normally.

Analysis

Why the messages appeared now

Commit 869d9e0e8 changed the matching in executors/docker/docker.go:

Before (19.1.1):

for _, pluggedContainer := range network.Containers { if id == pluggedContainer.Name {

After (19.2.0):

for containerID, pluggedContainer := range network.Containers { if id == containerID || id == pluggedContainer.Name {

network.Containers is a map keyed by container ID. The old code discarded the key and compared the incoming id argument against .Name only — a container ID compared against a container name, which never matched. As a result, NetworkDisconnect was never called and neither log line was ever reached.

With the fix, the match succeeds, the disconnect happens, and the existing log line surfaces on every job.

Why the level is wrong

The message sits in the else branch, i.e. the path taken when NetworkDisconnect returned no error:

err = e.dockerConn.NetworkDisconnect(ctx, netSummary.ID, id, true) if err != nil { e.BuildLogger.Warningln( "Can't disconnect possibly zombie container", pluggedContainer.Name, "from network", netSummary.Name, "->", err, ) } else { e.BuildLogger.Warningln( "Possibly zombie container", pluggedContainer.Name, "is disconnected from network", netSummary.Name, ) }

Both branches use Warningln, so a successful cleanup is reported with the same severity as a failed one.

Possible improvement

One option would be to log the success case at Info level and reword it, e.g.:

} else { e.BuildLogger.Infoln( "Container", pluggedContainer.Name, "disconnected from network", netSummary.Name, ) }

The failure branch would legitimately stay a WARNING. Happy to open an MR if this seems like the right approach.

Additional note

Since disconnectNetwork() was effectively a no-op before 19.2.0, its runtime behaviour has never been exercised in production. It now performs a NetworkList plus one NetworkInspect per network, per container cleaned up. On runners with many networks (e.g. FF_NETWORK_PER_BUILD enabled with high concurrency), this may be worth a look.

Environment

  • GitLab Runner 19.2.0 (previously 19.1.1, no warnings)
  • Executor: docker, talking to rootless Podman via unix:///run/user/<uid>/podman/podman.sock
  • Debian 12
  • Reproduced with FF_NETWORK_PER_BUILD both enabled and disabled (only the network name in the message changes)
  • Verified no actual leak: podman ps -a shows no surviving job containers, podman network ls shows no orphan networks, jobs succeed normally
Edited by Nolan Bedani