Retrospective: why the gateway-api v1.6.0 upgrade cost multiple days — and structural fixes for the next one

TL;DR

The dependency bump itself took under an hour. The days went to everything the bump's new, more aggressive conformance suite flushed out at once: five latent product bugs, a test-execution gap, four environment-capacity fragilities, and three process failures that multiplied CI round-trips. The upgrade was not expensive — it was the bill collector for debt that ordinary CI had never exercised. The structural fix is to stop letting that debt accumulate between upgrades.

Where the days actually went

Every CI round (fix → push → wait → diagnose) cost ~1h wall-clock; there were ~10 rounds plus local fresh-cluster verification cycles (~20 min each, ~15 cycles). The rounds were serial because each failure class hid the next one.

A. Latent product bugs exposed by the new suite's parallel churn (~40% of the time)

None of these were caused by v1.6.0 — its expanded suite (parallel tests, more Gateway churn, GatewayHTTPListenerIsolation etc.) was simply the first workload aggressive enough to trigger them:

  1. Event-pipeline family (coalescing starvation, bus drops under burst → mailbox rework, status-before-infrastructure ordering, stale-replay across leadership terms, deployedToPods startup race). One design weakness — best-effort event delivery with timing-dependent consumers — surfaced as five separate multi-hour investigations.
  2. Pod-port allocator read-back design: allocation stability depended on reading its own output back through the lagging informer cache. Under Gateway churn this produced a self-sustaining port collision (two Gateways on one bind; TLS alerts to plaintext) and a sustained Service-targetPort oscillation (252 flips/4s). Replaced with a pure hash — the single most serious product bug of the campaign, and it shipped invisible because nothing before the suite ever created/deleted Gateways in parallel.
  3. Assorted: coraza defaults silently DetectionOnly, load-path validationTests inheriting the live gate's 25s budget, startupProbe missing.

B. Tests that existed but ran nowhere (~15%)

The gateway e2e tests were profile-skipped locally and executed by no CI job — their first-ever execution was mid-upgrade, where their NodePort assumptions (stale since the per-Gateway-IP refactor) all failed at once and needed a port-forward migration with four sequential bug fixes. A test that never runs is worse than no test: it encodes false confidence and rots invisibly.

C. Environment/capacity fragility treated as flakiness (~25%)

Four distinct classes, each requiring full forensic diagnosis because early rounds lacked the logs to attribute them:

  • kube-proxy VIP DNAT-programming lag vs MetalLB's instant announce (freeze-proven; fixed via CONFORMANCE_PARALLEL=4 + upstream-default MaxTimeToConsistency)
  • reload-collision runtime-enable revert (#55 (closed), reopened with captured mechanism)
  • leader-election lease loss under runner starvation stalling deploys ~20s (#57 (closed))
  • unbounded test parallelism (GOMAXPROCS=16) vs actual runner capacity — the common amplifier behind all of the above

Half of the diagnosis time here was self-inflicted: the first rounds had no kube-proxy / MetalLB speaker logs in artifacts, so attribution was inference. Once those were added to after_script, later diagnoses took minutes instead of hours.

D. Process failures (~20%, the avoidable part)

  1. A fix commit that shipped only its CHANGELOG line — the values.yaml edit was never staged (c6714e15). Two rounds "passed by luck" before the crash-loop returned; nothing verified that the commit content matched its message.
  2. Fixing on a side branch and verifying the wrong ref — all fixes lived on fix-1178 while MR !1178 (merged)'s branch stayed unfixed for hours; "campaign complete, fully green" was reported while the user looked at an all-red MR pipeline.
  3. Two timing-dependent tests written during the campaign itself flaked later (mailbox burst test in round 1, coalesce-drain test on main post-merge) — both asserted exact scheduling outcomes instead of scheduling-independent contracts.

Structural improvements

Stop accumulating latent debt between upgrades

  1. Nightly conformance + e2e soak against main (scheduled pipeline, full suite, all shards). The event-pipeline and allocator bugs would each have surfaced individually, weeks apart, instead of as a 28-failure wall. This is the highest-leverage change.
  2. Pre-release canary job: a scheduled, allowed-to-fail job running the conformance suite from gateway-api's next RC/main against haptic main. Breakage becomes visible months before Renovate opens the bump MR.
  3. "Every test must run somewhere" gate: a CI check that fails when a test file matches no job's execution profile (would have caught the never-executed gateway e2e tests years earlier). Inventory: the compiled test binaries' -list output across build tags, compared against job run-patterns.

Make the environment a controlled variable

  1. Capacity right-sizing as policy, not incident response: parallelism knobs (PARALLEL, CONFORMANCE_PARALLEL) pinned to runner class in CI variables with the rationale documented; never default to GOMAXPROCS on shared runners.
  2. Budget doctrine: product-latency contracts (haptic convergence) live in dedicated e2e latency assertions; suite-level consistency budgets (MaxTimeToConsistency) stay at upstream defaults because they absorb cluster-infra latency haptic cannot control. This is now encoded in code comments — keep it that way in review.
  3. Diagnostics-first CI as a standing rule: every e2e/conformance job collects the full stack on failure (controller, HAProxy, DPA, kube-proxy, MetalLB, at-timeout config snapshots — all present now). New infra components join the after_script the day they're introduced.

Testing standards

  1. Scheduling-independence rule for concurrency tests (worth a CLAUDE.md bullet): tests may assert convergence contracts ("ends on latest, then quiescent"), never exact interleavings ("exactly one dispatch"); pace on observable state via introspection helpers, never sleeps. Both campaign flakes violated this identically.
  2. Churn/soak test tier: the worst bugs only appear under parallel create/delete churn. Add a deliberate churn test (N goroutines creating/deleting Gateways against a kind cluster for M minutes, asserting zero cross-wiring/oscillation via the debug allocator dump) so allocator-class regressions have a dedicated detector.

Process guardrails

  1. Commit-content verification: before declaring a fix pushed, diff the pushed commit against the intended change (not the commit message). Cheap habit; two full CI rounds were burned on the CHANGELOG-only commit.
  2. Deliverable-ref discipline: an upgrade campaign starts by writing down the MR whose head pipeline must go green; every fix lands on that branch; every "green" claim cites that pipeline ID.
  3. Upgrade playbook (docs/): the mechanical steps discovered this time — suite adaptation, CRD install, SupportedFeatures diff, shard rebalance, expected-new-tests review, timeout config policy — as a checklist so the next gateway-api bump starts from knowledge instead of archaeology.

Already done during this campaign (no further action)

  • Mailbox/coalescing rework, pure-hash allocator, status ordering, startupProbe, load-path budget, parallelism caps, full diagnostics collection, both flaky tests rewritten scheduling-independent.

Proposed next actions (each can become its own issue)

  • Nightly scheduled conformance+e2e pipeline against main (item 1)
  • gateway-api pre-release canary job (item 2)
  • "every test runs somewhere" CI gate (item 3)
  • Gateway churn/soak test (item 8)
  • Upgrade playbook doc (item 11)
  • CLAUDE.md testing-standards bullet: scheduling-independent assertions (item 7)