Bound render production by resource application completion
Summary
Backpressure the next render on completion of the current resource-application attempt. The coordinator's trigger/gate intake keeps draining while the resource applier performs live Kubernetes API operations.
ResourcesProcessedEvent acknowledges one authenticated render occurrence, including failed and gate-held attempts. A receipt for another occurrence cannot release the wait, even if its snapshot and output are identical. Only the existing ResourcesAppliedEvent authorizes rendered-status publication after successful convergence.
The current render and its HAProxy deployment remain asynchronous with resource application. Only the next render waits for the preceding resource attempt. Gate verdict ordering, live SSA, exact orphan-delete preconditions, forced-trigger priority, and context cancellation remain intact. No timer, queue enlargement, dropped verdict, local-payload apply skip, or weakened validation.
Reproduction and design
The preceding 5,000-route run (57bc9b46e, source 968e7bcfde7e) no longer overflowed the coordinator after !1769 (merged), but bulk teardown still produced six resource-applier mailbox warnings at depth 256. Render completions were interleaved with gate verdicts, preventing the applier's consecutive-only coalescing while it made API calls.
TestCoordinatorWaitsForResourceApplication fails on main f1b568e65 with real Make exit 2: a second render starts while the first resource attempt is pending. The correction permits at most one coordinator-produced resource cycle awaiting processing, while its intake drains a 4,096-trigger burst and retains the first forced trigger.
Failed and held attempts must acknowledge processing; otherwise the coordinator would deadlock the reconciliation that retries or repairs them. This acknowledgement is not a success verdict. ADR-0024 records the scheduling and validation contracts.
Verification
- Focused
make lint audit test: exit 0; 746 affected-package race tests plus 1,022 playground tests. Audit reports no reachable or imported-package vulnerabilities. - Full
make test: exit 0; 10,134 race-enabled tests with seven existing conditional skips in 818.933 s, plus 1,022 playground tests. - Fresh four-CPU HAProxy 3.4 E2E: exit 0 in 585.252 s; owned cluster removed. Running binary verified as source
98bf799eb98b. - Fresh archived-source 5,000-route scale run: runner and Make exit 0; all measured scenario gates passed. Whole-lifecycle audit: zero resource-applier backlog warnings, critical event drops, or controller iteration restarts. Owned cluster removed.
Scale lifecycle result
Both legs used fresh owned clusters, 50 namespaces × 100 routes, a 20-minute startup deadline, ten-minute steady churn, HAProxy 3.4, and no concurrent local builds or tests.
| Measurement | Before backpressure | With backpressure |
|---|---|---|
| Ramp to steady | 913.137 s | 894.876 s |
| Final ramp render p50/p90 | 62/91 ms | 68/87 ms |
| Final ramp deploy p50/p90 | 21/42 ms | 19/36 ms |
| Steady render p50/p90 | 42/56 ms | 41/53 ms |
| Steady deploy p50/p90 | 11/18 ms | 10/17 ms |
| Teardown resource-applier backlog warnings | 6 | 0 |
| Critical drops / iteration restarts | 0 / 0 | 0 / 0 |
The corrected run's final-ramp bucket contains 532 deployment events and its 600.009-second steady window contains 1,096. These are lifecycle measurements, not an isolated rendering-speed claim.
The whole-log audit also found one rejected render during ramp at 06:35:39 UTC, represented by three error log lines: render artifact "maps/backend-service.map" content differs from its plan file. The output consistency gate rejected the candidate before publication. This MR changes scheduling, not map construction or plan/artifact validation; the mismatch is being investigated separately and is not concealed as a warning-free run. No validation was bypassed.
Tests cover a blocked API call withholding its receipt, failed/held/follower processing without success publication, exact occurrence matching, stale acknowledgements, invalid identity, fanout isolation, cancellation at wakeup, and cleanup of the leader-term subscriptions. Existing gate release/revert and live-drift/recreation tests remain unchanged.
The full and E2E runs tested combined commit db8fa6def. After !1768 (merged) merged, this change was rebased onto actual main bbb480d0e as 133eb1f42; the complete tree diff is empty and its source hash remains 98bf799eb98b.