## Summary **Status update (2026-07-09):** follow-up investigation found measurement problems that invalidate parts of this description. Defect 2 is withdrawn (measurement artifact; fleeting does not use the go-plugin broker). Defect 3's local evidence is also contaminated (see strikethrough sections below), and a direct stress test of the taskscaler acquire path retains nothing meaningful. The production leak itself is real and confirmed against 62 days of fleet metrics; it remains unreproduced locally and un-root-caused. Full details in the [correction comment](https://gitlab.com/gitlab-org/gitlab-runner/-/work_items/39595#note_3539572500). Defect 1 was fixed via !6955. Runner managers running the docker-autoscaler executor with fleeting (AWS) leak anonymous heap memory at a rate proportional to job volume. On GitLab Dedicated hosted runner fleets we measured roughly 4 KB and 17 heap objects retained per successful job, which adds up to 160-280 MB/day on busy shards. Managers on 2 GB hosts eventually exhaust memory and wedge hard enough that AWS status checks fail (see incident refs below). ~~We reproduced the leak locally, profiled it, and traced three separate defects. One has a small fix ready (MR follows). The others need maintainer input.~~ Revised: only defect 1 held up. Defects 2 and 3 as originally described did not (see below). ## Measured behavior (production, GitLab Dedicated) This section still stands, it is metrics-based and unaffected by the profiling issues found later. Survey across multiple GitLab Dedicated hosted-runner fleets, 2026-07-07: - `AnonPages` grows linearly between restarts; `Cached` flat or shrinking. This is process heap, not page cache. - `go_memstats_heap_inuse_bytes` on the busiest shard: 21 MB after restart, 300 MB 5.5 days and 62k jobs later. That works out to ~4.5 KB/job. - `go_memstats_heap_objects`: ~17 objects retained per job. - `go_goroutines`: flat (~400). Not a goroutine leak in production. - Heap growth correlates with cumulative job count at r=0.93. Controlling for concurrency leaves the correlation intact (0.96); controlling for job count removes the concurrency signal. The leak follows jobs, not load. - Affected versions: 19.0.1, 19.0.2, 19.1.0 (entire surveyed fleet). <details><summary>Fleet survey data and slope table</summary> (attach: survey-7d, history-42d, analyze_dhr_mem.py output) Leak rates by shard: 160 to 280 MB/day (busiest, ~11k jobs/day), 30-55 MB/day mid-tier, 1-15 MB/day quiet shards. At least one full memory-exhaustion wedge and several near-misses (<300 MB available) observed over a multi-week window. </details> **Update (2026-07-09):** a follow-up 62-day analysis across three DHR shards (29 restart-bounded segments) confirms growth tracks cumulative jobs rather than time (median R2 0.86, beats the time fit in 27 of 29 segments), and the KB/job slope varies from 2.1 to 4.2 across shards, so the leak depends on job content or mix, not just count. Details in the correction comment linked above. ## Local reproduction Compose rig: runner 19.1.0, docker-autoscaler, fleeting AWS plugin against a moto stub, max_use_count=1, driven by backbench. (Rig: gitlab-com/gl-infra/ci-runners/sandbox/gitlab-runner-fleeting-aws-leak-repro) ~~Key result: job *content* matters.~~ **Revised (2026-07-09): no local configuration reproduces the leak.** With corrected methodology (gc=1 heap snapshots, coordinator kept alive during drain), trivial jobs (23,500), content-heavy jobs (8,937), content-diverse jobs (5,709), and high-concurrency runs (600 jobs at concurrency 50) all show a flat quiesced heap floor. The 14 KB/job figure below came from a 927-job run affected by two measurement artifacts: stale (non-gc) heap profiles, and stopping the fake coordinator, which parks in-flight jobs in a 10-17 minute final-update retry loop that mimics retention. Full writeup in the correction comment. ~~- 23,500 trivial jobs (`echo hello`, no artifacts): no measurable leak. Quiesced heap flat. - 927 jobs with ~300 KB trace output + 100 KB artifacts + several variables: heap 14.7 -> 27.4 MB quiesced, 39.8k -> 76.4k objects. ~14 KB and ~40 objects per job. Goroutines drained fully back to baseline (61), matching the production signature. - After 40 further minutes of quiescence, part of the retention drains slowly (contexts, decoded JSON strings) but ~10 MB / 26k objects remain.~~ <details><summary>pprof retention chains (quiesced heap diff) -- superseded, see note</summary> **Note: these profiles were captured without `gc=1` and during coordinator-stopped quiesce. Treat the numbers below as unreliable; kept for history only.** Top retained object stacks after 927 fat jobs, baseline-diffed, fully quiesced: 1. `context.(*cancelCtx).Done`: 4,681 objects. Allocation site: `context.WithTimeout` in `executors/internal/autoscaler/(*executor).Prepare` (executor.go:53). The context is passed into `taskscaler.Acquire`; retention survives job completion and drains only tens of minutes later. The deferred cancel() runs but something in the acquire path holds the context object. 2. `go-plugin (*gRPCBrokerStartStreamClient).Recv`: 5,461 objects, never drains. hashicorp/go-plugin broker stream receive buffers on the long-lived fleeting plugin connection, growing with traffic. 3. `encoding/json.(*decodeState).literalStore`: 38k objects at first quiesce, partially drains. Decoded job-payload strings pinned by holders above. Absent from all diffs: prometheus metric growth, trace buffers, http transports. Goroutines flat throughout. </details> ## Defect 1 (fixed): connectDocker orphans the SSH tunnel on error **Status: fixed via !6955.** `executors/docker/docker.go` `connectDocker()`: the freshly created `dockerConnection` is only assigned to `e.dockerConn` after `Info()`, `ServerVersion()`, version parse, and `validateOSType()` all succeed. If any of them fails, the connection (which for autoscaler executors wraps a live SSH tunnel to the instance) is orphaned. `Cleanup()` closes `e.dockerConn`, which still points at the previous (nil) connection. The tunnel's goroutines, bufio buffers and cipher state stay alive for the life of the process with nothing left holding a reference that could close them. Preparation retries multiply this by 3 per job. Local repro of the failure path (SSH up, docker daemon down): 1,446 complete leaked SSH clients after ~480 failed jobs, six goroutines each (mux loop, kex loop, channel handler, conn reader, client waiter, fleeting connector keepalive). These OOM-killed the 8 GB test VM in 17 minutes at 100% failure rate. Production note: this defect needs `connectDocker`-phase failures to fire. On our fleet those are rare, but any fleet with intermittent instance boot problems hits this hard, and it converts a transient failure into permanent memory loss. ## Defect 2 (withdrawn): go-plugin broker stream buffers accumulate ~~The fleeting plugin connection is process-lifetime. Its go-plugin gRPC broker stream (`newGRPCClient.func1` -> `StartStream` -> `Recv`) retains receive-message buffers that grow with traffic and never free. ~6 objects per job in our repro. We have not traced whether this is fixable in gitlab-runner, fleeting, or upstream hashicorp/go-plugin.~~ **Withdrawn (2026-07-09).** Two findings: the never-draining objects were an artifact of heap profiles taken without forced GC, and a code audit shows fleeting never uses the go-plugin broker: `internal/plugin/plugin.go` accepts the broker parameter and ignores it, so there are no per-job brokered streams to accumulate. No action needed in gitlab-runner, fleeting, or go-plugin. ## Defect 3 (revised): acquire-path context retention ~~Contexts created per job in `(*executor).Prepare` (executor.go:53) are retained past job completion despite the deferred cancel, draining only after tens of minutes. Retention path runs through `taskscaler.Acquire` / `Instance.Heartbeat` / singleflight / `CancellableGroupContext.WithContext` (which spawns a goroutine per derived context selecting on the group's done channel). Slow-drain retention at this volume behaves like a leak on small hosts: it raises the floor faster than it decays under continuous job flow.~~ **Revised (2026-07-09).** The slow-drain evidence was contaminated by the coordinator-stopped artifact: killing the job driver parks every in-flight job's final status update in a 10-17 minute retry loop (`network/trace.go` `finishWithFinalUpdate`), which holds trace buffers and Build graphs and looks exactly like acquire-path retention. A standalone taskscaler stress test (5,000 acquire/release/heartbeat cycles at concurrency 50) retains ~170 B / 0.36 objects per cycle, 25x below the production signature, and `buildsHelper`, the acquisitions map, and the acquisition ref all release synchronously. There is no evidence this path causes the production leak. Migrating `CancellableGroupContext` to `context.AfterFunc` is still worthwhile hygiene independent of this issue. ## Open questions for maintainers ~~- TODO: root-cause defect 2 (broker stream). Is the broker even needed for the fleeting plugin protocol? - TODO: root-cause defect 3's exact holder; decide whether CancellableGroupContext should use context.AfterFunc (Go 1.21+) instead of a goroutine per derived context.~~ - **TODO (primary open action): capture a `gc=1` heap profile from a busy production manager.** With 200+ MB of accumulated growth, the profile should name the holder chain directly. Capture plan in the correction comment. - Optional hygiene: migrate CancellableGroupContext to context.AfterFunc (not believed to fix this issue). - The unexplained fleet observation about deploy-gap slopes stands; the 62-day follow-up analysis found no restart segment longer than ~14 days, so the original 14-21-day contradiction could not be tested directly. ## Impact Any docker-autoscaler or instance-executor deployment with sustained job volume is affected, especially small manager hosts and max_use_count=1 fleets. On 2 GB hosts the manager eventually exhausts memory and the host wedges: kernel memory starvation, failed cloud status checks, and loss of remote management. Recovery requires a manager restart. This has caused production incidents on GitLab Dedicated hosted runners where managers wedged after memory exhaustion (INC-11727, incident-management#3368). A GitLab-internal mitigation and resilience plan tracks the recovery side separately; root cause remains open pending a production heap profile.