## Summary `TestIngressRollingRestartZeroDowntime` intermittently 503s under the full parallel e2e suite. The render→deploy→fast-path pipeline converges in **~46ms** normally (measured on CI [3.3]), but a failing CI run showed the new pod's server slot not reaching HAProxy for **~1.2s** after the pod went EndpointSlice-Ready. The ~1.2s is **not yet explained** and this issue tracks pinning + fixing it. This is **not** caused by the code-cleanup MR !1062 (separately verified: the deployer / dataplane sync / comparator / pipeline are untouched; the parser refactor is behavior-preserving; the version split is a timing-flake signature). ## Symptom - Single-replica backend rolling restart. A probe 503s in the transition window. - Two observed shapes of the same window: - **CI:** `SRV_1` (old, terminating) `SC--` connect-fail, new pod's `SRV_2` not yet enabled → no usable server. - **Local full-suite:** `SRV_2` (new pod) enabled promptly but connect timed out (`sC--`) — a node-overload artifact (reachability probe showed **1.55s** connect to a healthy pod). Different shape; not the CI cause. ## What's established (measured) - Normal convergence (CI [3.3], passing): new pod EPS-Ready → render emits its slot → fast-path applies in **~46ms** (e.g. Ready 12:08:16.202 → emit 16.248). - Failing CI run: ~1.2s gap (Ready 56.634 → first emit 57.899). During the gap, **16 render+validate cycles completed normally** (~80ms each, `render_ms` 65–130, no stalls/gaps, node 30% CPU), yet none emitted the new server (zero fast-path applies; deploy baseline never advanced). ## Ruled out - **Store-read staleness** — proven impossible by code audit: watcher `store.Update` is synchronous/same-goroutine (`pkg/k8s/watcher/watcher_handlers.go:147-179`), `MemoryStore` mutates under `RWMutex` (happens-before), reads are per-render-fresh (`pkg/controller/rendercontext/builder.go:468-477` build fresh wrappers; eager `Store.List()` at first access `store_wrapper.go:202`). - **Deployer baseline/diff** — `lastDispatchedParsed` advances only adjacent to a logged apply (`scheduler_logic.go:337` / `:359`); `applyRuntimePreInterval` deliberately does not advance it; so `ServerOpCount=0` means the render's `ServerIndex` genuinely lacked the new server. - **Watcher commit timing** — synchronous; debouncer never touches the store write. - **Coordinator stall / slow renders / CPU starvation / on-demand-cache staleness / lane-guard / render-logic** — all excluded by measurement or audit. ## Remaining hypotheses (need a captured failing run to split) - **(A) — most likely — committed EndpointSlice content.** A single-replica rolling restart writes the slice several times in rapid succession (new pod added not-ready → flips ready/serving → old pod terminating), often coalesced. If the renders in the gap read a committed shape where the new pod was still `Ready==false`, the chart correctly filtered it out (`charts/haptic/libraries/base.yaml:567`), and the T0+1.2s emission is the genuinely-ready write arriving. This is the only hypothesis fully consistent with every verified code path. NOTE: in the analyzed CI run the raw `watcher update` payload at T0 showed the new pod `ready:true`, so confirming (A) requires checking the **converted/stored** shape and the per-render readiness evaluation, not just the raw watch payload. - **(B) store-keying mismatch.** Eager-mode exact-match `Fetch(ns, serviceName)` returns empty on a key miss with no fallback (`store_wrapper.go:368-373`). If the committed slice's `kubernetes.io/service-name` label diverged from the fetch key, the render never saw it. Less likely (index is spec-standard) but cheap to rule out. ## Next step (how to confirm) 1. Re-apply the per-render diagnostic (preserved in closed MR !1064, commit `ba49b914`): logs per render `lane` / `ServerOpCount` / `StructuralOpCount` / `summarizeActiveServers(parsedConfig)`. 2. Add ONE readiness-evaluation log inside `BackendServers` (`base.yaml:566-572`): per render, for the rolling-restart service, `len(allSlices)` and each endpoint's `addr` + `Conditions.Ready` (nil vs *value) + `Conditions.Terminating`. 3. Set the e2e artifact rule to `when: always` (excl. pcaps) on the diagnostic branch so a passing run also captures the data. 4. Capture a run (flake or pass) and read the rolling-restart window: - `len(allSlices)>0` but new IP shows `Ready==false` across the gap, flipping at T0+1.2s → **(A)**, watcher-content/condition timing. - new pod's slice absent from `allSlices` → **(B)**, keying/routing. ## Fix direction (once confirmed) The premise is fast convergence (other controllers avoid 503s by converging in ms; HAPTIC already does ~46ms normally). The fix targets whatever pins the slow tail — likely either how rolling-restart slice-condition transitions are observed/coalesced (A) or endpoint keying (B). Do NOT pursue: per-server runtime endpoint (flaps on reload), slot reuse (breaks redispatch), preStop/graceful-shutdown changes to user pods (wrong layer). ## Acceptance `TestIngressRollingRestartZeroDowntime` passes reliably under the full parallel suite across all HAProxy versions, with the root cause of the ~1.2s tail confirmed by a captured failing/slow run.