sylvactl reports false "narrow timeouts" — reconcileCompletedAt annotation dropped for fast units

The analysis and resume below were done by Claude Opus 4.8, so feel free to challenge it. I am backing it up with the fact that I was running a CI test with SYLVA_TOOLBOX v1.9.0, which contains sylvactl v0.14.8 and the test_narrow_unit_timeouts CI tests were fine (see https://gitlab.com/sylva-projects/sylva-core/-/pipelines/2692727016 for reference)...so we indeed introduced a regression between this and sylvactl v0.15.1.

Since v0.15.1 (regressing MR !589 (merged) / commit b487bbb), the narrow-timeout check reports healthy units as taking ~1000s to reconcile. sylvactl fails to write its sylvactl/reconcileCompletedAt.<gen>.<runID> annotation for units that reach Ready quickly, so the duration is computed as now − start instead of end − start.

After a sylva-units re-release ~all fast units get flagged at once:

AssertionError: Detected narrow timeouts:
 - Kustomization/ceph-csi-rbd took 1055.69s to reconcile (target 240.0s ...)
 - ... (17 units)

The units are healthy on the cluster. Their annotations show the tell: start present, completion missing.

sylvactl/reconcileStartedAt.2.AIC2D7:   2026-07-20T12:49:35   # present
# sylvactl/reconcileCompletedAt.2.AIC2D7                       # MISSING
# status: Ready=True, observedGeneration=2, reconciled in ~11s

Missing completion → ReconcileEnd() returns zero time → CurrentReconcileDuration() falls back to time.Since(start) ≈ 1000s → ComputeTimeoutInfo flags it.

The completion is written only on the not-ready→ready edge, guarded on the start annotation being visible on the in-memory object:

// handleEvent
if obj.Ready() && (prev == nil || !prev.Ready()) {
    t.setReconcileCompletedAnnotations(obj)
}
// setReconcileCompletedAnnotations
if object.GetAnnotationValue(ReconcileStartedAnnotation+reconcileID) != "" && ... {

But setStartReconcile persists the start annotation asynchronously (go t.UpdateObject, patching a copy on the API server — it does not mutate the in-memory obj). The annotation only appears on obj after it round-trips through the watch.

The race, per unit:

  1. Not-ready at gen N → setStartReconcile launches the async reconcileStartedAt.N patch; obj doesn't carry it yet.
  2. Unit reaches Ready before that patch is observed back → the edge fires but the guard is false → completion skipped.
  3. The "preserve previous ready" branch keeps prev ready, so when reconcileStartedAt.N finally lands the edge never re-fires → reconcileCompletedAt.N is permanently lost.

MR !589 (merged) comments out the !HasHandledReconcileAnnotation() gate in computeStatus. That gate used to hold objects not-ready for the whole reconcile request→handled window — long enough for the async start-annotation patch to be observed before the ready transition. Removing it lets fast units flip to Ready immediately, exposing the pre-existing async-visibility race.

cc @feleouet