Diff projected status snapshots by shared immutable roots

Summary

Diff authenticated projected-status snapshots through shared immutable target-index roots. Unchanged subtrees require no metadata replay; changed targets materialize only their current contributions, in the same order as full replay. Removed and reordered contributions correctly expose earlier projected or direct patches.

Failed status writes retain target identities, not obsolete payloads. The next event resolves retries against its own authenticated snapshot; removed targets and removed phases stop retrying. Existing lifecycle gates, resource-version checks, and phase-specific SSA ownership remain intact.

The generic persistent-tree diff is independent of Kubernetes resources. Only the projection lineage index changes representation; group and phase indexes keep their existing representation. Independently built trees fall back to exact comparison, and structurally shared trees skip identical subtrees without keeping a revision history chain.

Measurements

Same-machine, back-to-back native Go benchmarks, six 200 ms samples per case, medians. Before is 5d880a32e; after is the implementation committed as da0c7f9a7. No concurrent local tests or builds during these measurements.

Operation	Before	After	Allocation bytes before → after
One projected target changed, 300 targets	168.8 µs	2.21 µs	110,904 → 2,264
One projected target changed, 3,000 targets	1,560.4 µs	1.88 µs	856,648 → 2,264
One projected target changed, 10,000 targets	5,431.0 µs	1.83 µs	3,413,512 → 2,264
One change in a grown 3,000-target index	1,948.7 µs	2.13 µs	856,649 → 2,264
Replace one projection-plan entry of 3,000	8.72 µs	9.14 µs	14,104 → 14,552

The selected-target operation stays flat in the measured fleet sizes; this is not a claim that whole rendering takes 2 µs. Full projected materialization remains about 2.4 ms, and detached snapshots still require full comparison. Maintaining the target index costs approximately 4.8% more in the one-entry replacement microbenchmark (448 additional bytes and three allocations).

The matched 5,000-route scale leg compares the allocation-only leg (f86235cee, source 144daa0b2ef1) with an archived measurement commit containing this status implementation on the same base (dddea1b20, source 61fd95c11c42). Implementation files match this branch exactly; the intervening main-branch determinism fix is excluded from both measurement legs and included in product validation. Both legs use fresh owned clusters, HAProxy 3.4, 50 namespaces with 100 routes each, and 600-second steady windows without competing local builds or tests.

Fleet measurement	Allocation baseline	Status delta
Ramp to 5,000 routes	974.618 s	939.707 s
Final-ramp render p50 / p90	72 / 105.1 ms	72 / 94 ms
Final-ramp deploy p50 / p90	22 / 39.1 ms	20 / 37.8 ms
Steady render p50 / p90	42 / 56 ms	42 / 56 ms
Steady deploy p50 / p90	11 / 18 ms	10 / 18 ms
Steady total p50 / p90	59 / 111 ms	59 / 108 ms

Final ramp covers 4,500–5,000 routes (530 versus 513 deployment events); steady samples are 1,051 versus 1,075. Quantiles use linear interpolation after deduplicating event IDs and joining timestamps to the latest route census. Steady render and total medians are unchanged; the main gain is removal of fleet-wide status work, not another large whole-render improvement.

Matched approximately 4,500-route, 20-second leader profiles show status-applier cumulative CPU falling from about 3.4 s to 0.1 s, with total CPU 33.18 → 31.63 s. Sampled allocation increased from 11,614.57 to 12,379.39 MiB (20.32 versus 20.28 s windows); GC marking increased from 11.66 to 12.54 CPU-seconds. These wall-window profiles are not normalized per render and do not establish a total-allocation improvement for this change.

Teardown finding and separate correction

The runner exited 0 and all measured-scenario gates passed, but controller logs revealed a failure after the steady window ended at 20:24:23 UTC: bulk teardown overflowed the reconciliation-coordinator subscription at 20:24:53, causing a fail-closed controller-iteration restart. The new leader also logged resource-applier backlog at 256 and 512 entries. No warnings or errors occurred during the measured window. This is a valid measured-window comparison, not a clean end-to-end lifecycle verdict. The coordinator intake defect is fixed separately in merged !1769 (merged); neither its buffer nor the critical-drop failure gate was relaxed.

The corrected combined run includes !1769 (merged) and the determinism fix, from clean archived commit 57bc9b46e / source 968e7bcfde7e. Runner exit 0; all measured gates passed; ramp 913.137 s, steady 600.012 s. The complete controller logs contain no errors, critical event drops, or iteration restart. Six resource-applier backlog warnings at depth 256 remain during bulk teardown (00:52:19–00:52:36 UTC). This establishes that the coordinator failure is fixed, but not a warning-free lifecycle. The resource-application scheduling redesign is approved as a separate MR and is not a dependency of this status improvement. This combined leg is functional verification, not an isolated performance comparison.

Verification

• make lint audit test: exit 0; full race-enabled suite, 10,111 tests with seven existing conditional skips; playground suite, 1,022 tests.
• Fresh-cluster full E2E on HAProxy 3.4 with controller.resources.limits.cpu=4: exit 0, main E2E package 553.136 s; owned cluster removed.
• Old-code regression run: 13 tests, eight expected failures. It reproduces removed/reordered contribution and stale retry defects; the same regressions pass in the new full suite.
• Randomized branched-snapshot differential tests, frozen and grown 10,000-key sharing checks, hash collisions, tombstones/restores, independent roots, and early termination.
• Snapshot, plan, group, source, and composite-leaf authentication tests; phase removal, skipped snapshots, output detachment, direct fallback, and current-revision retry coverage.
• Corrected combined tree: lint/audit exit 0; 10,117 full race-enabled tests with seven existing conditional skips, plus 1,022 playground tests; fresh four-CPU HAProxy 3.4 E2E exit 0 in 551.236 s. After !1769 (merged) merged, this branch was rebased onto merge f1b568e65 as 013cd7880. Its complete file tree is identical to tested 57bc9b46e, and the source hash remains 968e7bcfde7e.

Root-template enumeration remains a separate chart/engine design in #212 (closed). The scale-run publication-status conflict is outside this change; no validation or retry gate is weakened here.