Incremental output rejects a map whose artifact differs from its plan file
Observed failure
One reconciliation failed during the ramp of a fresh 5,000-route scale run on 2026-09-10 at 06:35:39.907 UTC:
render failed: sealing rendered output: committing incremental output publication: render artifact "maps/backend-service.map" content differs from its plan fileThe error appears in three log lines for one failed render, correlation c9f8a2718ddc3372e39afde1515255f5. The consistency gate rejected the candidate before publication; a later reconciliation succeeded. This is not three separate failures and does not establish that inconsistent output reached HAProxy.
Source and workload
- Archived commit
db8fa6defe8ca990ebf8195bee9c7c92da4b8c91, source hash98bf799eb98b; complete tree identical to !1770 (merged) revision133eb1f42bd55d665ade42d1129487804a464172. - Fresh owned Kind cluster, HAProxy 3.4, 50 namespaces × 100 routes, 20-minute startup deadline, ten-minute steady churn, no concurrent local builds or tests.
- Runner and Make exited 0, all measured gates passed. Whole-controller-log audit exposed this rejected render. No backlog warnings, critical event drops, or iteration restarts occurred.
- Ramp 894.876 seconds; steady 600.009 seconds. The error occurred during ramp, not teardown.
Investigation
RenderService.finishRender constructs the plan and artifact candidates from the same auxiliary file set. The rejection originates in renderoutput.validateArtifactContent during incremental output commit. The cause is not established.
Three race-enabled focused probes passed without reproducing it: 2,000 simple map transitions; deferred plan materialization with concurrent previous-output reads; and 2,000 mixed full/incremental publications with 12 independently changing maps. Passing these probes does not invalidate the observed failure.
A local diagnostic-only revision adds failure-path comparisons of current inputs, candidate plan files, artifacts, delta files, and previous artifacts. It logs paths, lengths, and equality booleans only, not file contents, and still returns the original rejection. This is diagnosis, not a runtime fix.
The instrumented run completed on 2026-09-10 using archived local commit 852a82167a0df798baf95496e3423dee2b539b83, source hash 062636ba9bc5, and the same fresh-cluster 50 × 100 workload. It reached 5,000 routes at 07:29:51 UTC and completed ten minutes of steady churn plus teardown. Make and the harness exited 0, and all measured gates passed. The owned cluster was removed.
Whole-controller-log inspection found no map mismatch, diagnostic failure, backlog warning, critical event drop, or controller error. One distinct auxiliary-reference status update conflict occurred at 07:39:22.425 UTC; this was not a warning-free run. A supplemental live log stream ended with EOF when the cluster was removed; the harness captured the complete scenario log separately.
The rejection did not reproduce, so the failure-path diagnostics captured no mismatching candidate. The diagnostic commit remains local and has not been pushed or merged. No root cause or runtime fix is established. Do not close this issue on the strength of the passing run.
Required result
Capture the mismatching candidate, name the mechanism, add a failing regression, and fix it without relaxing output consistency or publishing a fallback after a rejected incremental candidate. Verify affected tests, the full local gates, and a fresh scale lifecycle. Keep this issue open until that evidence exists.