Runner manager leaks heap memory proportional to job count with docker-autoscaler executor

Summary

Status update (2026-07-09): follow-up investigation found measurement problems that invalidate parts of this description. Defect 2 is withdrawn (measurement artifact; fleeting does not use the go-plugin broker). Defect 3's local evidence is also contaminated (see strikethrough sections below), and a direct stress test of the taskscaler acquire path retains nothing meaningful. The production leak itself is real and confirmed against 62 days of fleet metrics; it remains unreproduced locally and un-root-caused. Full details in the correction comment. Defect 1 was fixed via !6955.

Runner managers running the docker-autoscaler executor with fleeting (AWS) leak anonymous heap memory at a rate proportional to job volume. On GitLab Dedicated hosted runner fleets we measured roughly 4 KB and 17 heap objects retained per successful job, which adds up to 160-280 MB/day on busy shards. Managers on 2 GB hosts eventually exhaust memory and wedge hard enough that AWS status checks fail (see incident refs below).

We reproduced the leak locally, profiled it, and traced three separate defects. One has a small fix ready (MR follows). The others need maintainer input. Revised: only defect 1 held up. Defects 2 and 3 as originally described did not (see below).

Measured behavior (production, GitLab Dedicated)

This section still stands, it is metrics-based and unaffected by the profiling issues found later.

Survey across multiple GitLab Dedicated hosted-runner fleets, 2026-07-07:

  • AnonPages grows linearly between restarts; Cached flat or shrinking. This is process heap, not page cache.
  • go_memstats_heap_inuse_bytes on the busiest shard: 21 MB after restart, 300 MB 5.5 days and 62k jobs later. That works out to ~4.5 KB/job.
  • go_memstats_heap_objects: ~17 objects retained per job.
  • go_goroutines: flat (~400). Not a goroutine leak in production.
  • Heap growth correlates with cumulative job count at r=0.93. Controlling for concurrency leaves the correlation intact (0.96); controlling for job count removes the concurrency signal. The leak follows jobs, not load.
  • Affected versions: 19.0.1, 19.0.2, 19.1.0 (entire surveyed fleet).
Fleet survey data and slope table (attach: survey-7d, history-42d, analyze_dhr_mem.py output) Leak rates by shard: 160 to 280 MB/day (busiest, ~11k jobs/day), 30-55 MB/day mid-tier, 1-15 MB/day quiet shards. At least one full memory-exhaustion wedge and several near-misses (<300 MB available) observed over a multi-week window.

Update (2026-07-09): a follow-up 62-day analysis across three DHR shards (29 restart-bounded segments) confirms growth tracks cumulative jobs rather than time (median R2 0.86, beats the time fit in 27 of 29 segments), and the KB/job slope varies from 2.1 to 4.2 across shards, so the leak depends on job content or mix, not just count. Details in the correction comment linked above.

Local reproduction

Compose rig: runner 19.1.0, docker-autoscaler, fleeting AWS plugin against a moto stub, max_use_count=1, driven by backbench. (Rig: gitlab-com/gl-infra/ci-runners/sandbox/gitlab-runner-fleeting-aws-leak-repro)

Key result: job content matters.

Revised (2026-07-09): no local configuration reproduces the leak. With corrected methodology (gc=1 heap snapshots, coordinator kept alive during drain), trivial jobs (23,500), content-heavy jobs (8,937), content-diverse jobs (5,709), and high-concurrency runs (600 jobs at concurrency 50) all show a flat quiesced heap floor. The 14 KB/job figure below came from a 927-job run affected by two measurement artifacts: stale (non-gc) heap profiles, and stopping the fake coordinator, which parks in-flight jobs in a 10-17 minute final-update retry loop that mimics retention. Full writeup in the correction comment.

~~- 23,500 trivial jobs (echo hello, no artifacts): no measurable leak. Quiesced heap flat.

  • 927 jobs with ~300 KB trace output + 100 KB artifacts + several variables: heap 14.7 -> 27.4 MB quiesced, 39.8k -> 76.4k objects. ~14 KB and ~40 objects per job. Goroutines drained fully back to baseline (61), matching the production signature.
  • After 40 further minutes of quiescence, part of the retention drains slowly (contexts, decoded JSON strings) but ~10 MB / 26k objects remain.~~
pprof retention chains (quiesced heap diff) -- superseded, see note

Note: these profiles were captured without gc=1 and during coordinator-stopped quiesce. Treat the numbers below as unreliable; kept for history only.

Top retained object stacks after 927 fat jobs, baseline-diffed, fully quiesced:

  1. context.(*cancelCtx).Done: 4,681 objects. Allocation site: context.WithTimeout in executors/internal/autoscaler/(*executor).Prepare (executor.go:53). The context is passed into taskscaler.Acquire; retention survives job completion and drains only tens of minutes later. The deferred cancel() runs but something in the acquire path holds the context object.

  2. go-plugin (*gRPCBrokerStartStreamClient).Recv: 5,461 objects, never drains. hashicorp/go-plugin broker stream receive buffers on the long-lived fleeting plugin connection, growing with traffic.

  3. encoding/json.(*decodeState).literalStore: 38k objects at first quiesce, partially drains. Decoded job-payload strings pinned by holders above.

Absent from all diffs: prometheus metric growth, trace buffers, http transports. Goroutines flat throughout.

Defect 1 (fixed): connectDocker orphans the SSH tunnel on error

Status: fixed via !6955.

executors/docker/docker.go connectDocker(): the freshly created dockerConnection is only assigned to e.dockerConn after Info(), ServerVersion(), version parse, and validateOSType() all succeed. If any of them fails, the connection (which for autoscaler executors wraps a live SSH tunnel to the instance) is orphaned. Cleanup() closes e.dockerConn, which still points at the previous (nil) connection. The tunnel's goroutines, bufio buffers and cipher state stay alive for the life of the process with nothing left holding a reference that could close them. Preparation retries multiply this by 3 per job.

Local repro of the failure path (SSH up, docker daemon down): 1,446 complete leaked SSH clients after ~480 failed jobs, six goroutines each (mux loop, kex loop, channel handler, conn reader, client waiter, fleeting connector keepalive). These OOM-killed the 8 GB test VM in 17 minutes at 100% failure rate.

Production note: this defect needs connectDocker-phase failures to fire. On our fleet those are rare, but any fleet with intermittent instance boot problems hits this hard, and it converts a transient failure into permanent memory loss.

Defect 2 (withdrawn): go-plugin broker stream buffers accumulate

~~The fleeting plugin connection is process-lifetime. Its go-plugin gRPC broker stream (newGRPCClient.func1 -> StartStream -> Recv) retains receive-message buffers that grow with traffic and never free. ~6 objects per job in our repro. We have not traced whether this is fixable in gitlab-runner, fleeting, or upstream hashicorp/go-plugin.~~

Withdrawn (2026-07-09). Two findings: the never-draining objects were an artifact of heap profiles taken without forced GC, and a code audit shows fleeting never uses the go-plugin broker: internal/plugin/plugin.go accepts the broker parameter and ignores it, so there are no per-job brokered streams to accumulate. No action needed in gitlab-runner, fleeting, or go-plugin.

Defect 3 (revised): acquire-path context retention

Contexts created per job in (*executor).Prepare (executor.go:53) are retained past job completion despite the deferred cancel, draining only after tens of minutes. Retention path runs through taskscaler.Acquire / Instance.Heartbeat / singleflight / CancellableGroupContext.WithContext (which spawns a goroutine per derived context selecting on the group's done channel). Slow-drain retention at this volume behaves like a leak on small hosts: it raises the floor faster than it decays under continuous job flow.

Revised (2026-07-09). The slow-drain evidence was contaminated by the coordinator-stopped artifact: killing the job driver parks every in-flight job's final status update in a 10-17 minute retry loop (network/trace.go finishWithFinalUpdate), which holds trace buffers and Build graphs and looks exactly like acquire-path retention. A standalone taskscaler stress test (5,000 acquire/release/heartbeat cycles at concurrency 50) retains ~170 B / 0.36 objects per cycle, 25x below the production signature, and buildsHelper, the acquisitions map, and the acquisition ref all release synchronously. There is no evidence this path causes the production leak. Migrating CancellableGroupContext to context.AfterFunc is still worthwhile hygiene independent of this issue.

Open questions for maintainers

~~- TODO: root-cause defect 2 (broker stream). Is the broker even needed for the fleeting plugin protocol?

  • TODO: root-cause defect 3's exact holder; decide whether CancellableGroupContext should use context.AfterFunc (Go 1.21+) instead of a goroutine per derived context.~~

  • TODO (primary open action): capture a gc=1 heap profile from a busy production manager. With 200+ MB of accumulated growth, the profile should name the holder chain directly. Capture plan in the correction comment.

  • Optional hygiene: migrate CancellableGroupContext to context.AfterFunc (not believed to fix this issue).

  • The unexplained fleet observation about deploy-gap slopes stands; the 62-day follow-up analysis found no restart segment longer than ~14 days, so the original 14-21-day contradiction could not be tested directly.

Impact

Any docker-autoscaler or instance-executor deployment with sustained job volume is affected, especially small manager hosts and max_use_count=1 fleets. On 2 GB hosts the manager eventually exhausts memory and the host wedges: kernel memory starvation, failed cloud status checks, and loss of remote management. Recovery requires a manager restart.

This has caused production incidents on GitLab Dedicated hosted runners where managers wedged after memory exhaustion (INC-11727, incident-management#3368). A GitLab-internal mitigation and resilience plan tracks the recovery side separately; root cause remains open pending a production heap profile.

Edited by Kam Kyrala