feat(virtual): resolution phase machine (S13 Step 12, part 2/2)

David Fernandezrequested to merge
10io/s13-step-12-virtual-resolution-part2of2 into main

📚 Stacked MRs

This change is split into 2 stacked MRs. Each part targets the previous one (part 1 targets main); review and merge proceed bottom-up. This part deliberately exceeds the review-size guideline — see the waiver below.

📦 What this MR does

Part 2 of 2 of S13 virtual/remote foundation — Step 12: virtual resolution: the network-bound resolution phase machine composing the S13 foundation operations over part 1's decision core.

  • Resolver.Resolve (Virtual resolution): Phase 0 allow/deny filtering with denial audit events and the unhealthy-remote skip; Phase 1 parallel Lookup with the fresh-hit short-circuit; Phase 2 bounded parallel HEAD fan-out with abort-on-winner; Phase 3 winner dispatch.
  • Winner dispatch: cache serve, 304 revalidation bump (detached, with the vanished-row full re-fetch), and the coalesced fetch honoring the winner fetch failure no-fall-through contract: a transport failure returned by the fetch call takes the cache fallback, while a failure after the stream has begun propagates to the caller and leaves the cached row as it was.
  • Terminal and infrastructure errors: ErrNoUpstreams, ErrAllUpstreamsDenied, ErrAllUpstreamsUnhealthy (literal: every rule-eligible upstream is an unhealthy remote), ErrFileNotFoundOnUpstreams only when some position established an absence (a probe 404 or a hosted miss) — an outage across every upstream serves a cached copy if one exists and otherwise returns ErrUpstreamUnavailable, per no winner; Lookup/health-read errors abort loudly rather than degrade positionally; caller abandonment surfaces ErrResolverCancelled; ErrLeaderCanceled is retried exactly once; ErrSingleFlightTimeout and ErrHeaderInjection propagate and are never fallback-eligible.
  • Seam additions in internal/remote: FallbackEligible, the closed list the cache fallback opts into (with ErrLeaderPanicked exported, since a recovered leader panic is eligible while a stub seam's ErrNotImplemented propagates); the UpstreamDenialAuditSink interface beside its event payload — which carries Phase and Reason (ExclusionPolicyDenied or ExclusionInvalidRule), so an invalid rule is audited as itself rather than read as a deny-rule match — its remotetest fake, and BumpUpstreamCheckedAt, which owns the detached 304 freshness write both the fetch pipeline and the resolver's ProbeFresh winner run through (the timeout it bounds stays unexported).
  • The composed-flow coalescing guarantee (N+1 HEADs, exactly one upstream GET, one cache upsert under concurrent resolutions) is proven at unit level and against the real BlobStore/Postgres two-phase commit (integration), with internal/virtual wired into the test:integration CI job.

⚠️ Review-size waiver

This MR is ~2,460 reviewable LoC, above the 1,000 guideline. It ships whole deliberately:

  • Production code is 575 LoC (resolve.go); the rest is its test suite, dominated by resolve_test.go (~1,660 LoC in this part).
  • The resolver exposes a single entry point (Resolve), and both test files are external-package (virtual_test), so every test reaches production code only through Resolve. Splitting the suite away from resolve.go would land a concurrent state machine with partial coverage and defer the remaining evidence to a test-only follow-up — strictly worse for review than size.
  • The test volume reflects the resolver's state space (phases × lookup/probe outcomes × abandonment points × fetch error classes) plus the barrier choreography that makes the concurrency assertions race-free, not assertion verbosity.
  • The plan's step-12 "Size watch" note anticipated exactly this cut: the pure decision core first (part 1), the indivisible network phase machine second (this MR).

✅ Spec coverage

Requirement Tests
Virtual scenarios 1–9: priority order, hosted wins, fresh-hit short-circuit, denial, all-stale winner 304, failure below winner irrelevant TestResolve_TwoRemotes_Position0Wins, TestResolve_TwoRemotes_Position0Misses, TestResolve_HostedWinsOverMissingRemote, TestResolve_RemoteBeatsHostedFreshHit, TestResolve_HostedShortCircuit_NoProbes, TestResolve_DeniedUpstream_AuditedAndNotContacted, TestResolve_AllStale_Winner304, TestResolve_FailureBelowWinner_Irrelevant, TestResolve_BlockedUpstream_ResolutionContinues, TestResolve_DigestKeyedHit_ServedWithoutRevalidation
Method discipline: N parallel HEADs + one GET on the winner; no body bytes from losers TestResolve_MethodDiscipline_NParallelHeadsOneGet, TestResolve_WinnerFetch_CarriesConditionalHeaderAndTemplate
Terminal cases: no upstreams, all denied, all unhealthy, not found anywhere TestResolve_NoUpstreams, TestResolve_AllUpstreamsDenied, TestResolve_AllUpstreamsUnhealthy, TestResolve_HostedMissWithAllRemotesUnhealthy_FileNotFound, TestResolve_NoWinnerAnywhere_FileNotFound
Unhealthy-remote skip without requests; invalid rules excluded; hosted denial audited with the zero repository UUID TestResolve_UnhealthyRemote_SkippedWithoutRequests, TestResolve_InvalidRule_ExcludedAndAuditedAsInvalidRule, TestResolve_DeniedHostedUpstream_AuditEventCarriesZeroRepositoryID
Winner fetch failure: no fall-through; cache fallback on a transport failure returned by the fetch call; a failure after the stream begins propagates (contract, not fallback); the fallback is opt-in through remote.FallbackEligible, so an unclassified error propagates TestResolve_WinnerFetchTransportFailure_ServesCacheFallback (fetch-call failure with a stale row), TestResolve_WinnerFetchTransportFailure_NoCacheEntry (no row, UpstreamUnavailable propagates), TestResolve_WinnerFetchNon2xx_WinnerFetchFailed
Infrastructure errors abort loudly; abandonment classification at every phase TestResolve_LookupError_AbortsLoudly, TestResolve_HealthReadError_AbortsLoudly, TestResolve_ProbeBuildError_AbortsLoudly, TestResolve_CallerCanceledMidPhase1Lookup_ResolverCancelled, TestResolve_CallerCanceledDuringHealthRead_ResolverCancelled, TestResolve_CallerCanceledDuringRevalidatedReRead_ResolverCancelled, TestResolve_CallerCanceledDuringFallbackRead_ResolverCancelled, TestResolve_CallerCanceledMidPhase2_ResolverCancelled, TestResolve_CallerCanceledBeforePhase2Probes_ResolverCancelled, TestResolve_CallerDeadlineMidPhase2_ResolverCancelledNotFileNotFound, TestResolve_CallerAbandonedMidPhase3_ResolverCancelled, TestResolve_WinnerFetchAbandonedInsideCacheFill_ResolverCancelled
No winner: an established absence is not-found; an outage serves cache or returns unavailable TestResolve_AllProbesTransportFailure_Unavailable, TestResolve_AllProbes5xx_Unavailable, TestResolve_AllProbesFailWithStaleRow_ServesCacheFallback, TestResolve_ProbeMissAlongsideProbeError_FileNotFound, TestResolve_HostedMissWithUnreachableRemote_FileNotFound
Cache fallback is a closed list, not a catch-all TestFallbackEligible (internal/remote), TestResolve_WinnerFetchNotImplemented_PropagatedNotFallback, TestResolve_WinnerFetchLeaderPanicked_ServesCacheFallback
Resolution edge cases: 304 races, vanished rows, fetch error classes, leader-canceled retry TestResolve_WinnerFetch304Race_ServesCurrentCacheRow, TestResolve_ProbeFreshWinner_VanishedRow_FetchesInFull, TestResolve_ProbeFreshWinner_RowReplacedAfterLookup_ServesCurrentRow, TestResolve_ProbeFreshWinner_RowVanishedAfterBump_ErrCacheFill, TestResolve_ProbeFreshWinner_BumpInfraError_ErrCacheFill, TestResolve_WinnerFetch304Race_ServesCurrentCacheRow (re-read failure and vanished row both ErrCacheFill), TestResolve_WinnerFetchHeaderInjection_PropagatedNotFallback, TestResolve_WinnerFetchCacheFillError_PropagatedNotFallback, TestResolve_WinnerFetchSingleFlightTimeout_Propagated, TestResolve_WinnerFetchLeaderCanceled_RetriedOnce, TestResolve_WinnerFetchLeaderCanceledTwice_Propagated
Composed-flow coalescing (unit + real DB) TestResolve_ConcurrentResolutions_CoalesceToOneUpstreamGET (unit), TestResolve_Integration_ConcurrentResolutions_CoalesceToOneUpstreamGET (integration)
Constructor validation and zero-value guards TestNewResolver_Validation, TestResolve_ZeroValueUpstreamSpec_Guarded

🔍 Notes for reviewers

  • The denial-audit sink and fake mirror the existing health-audit pair in internal/remote; the fake's suite gains the mirrored recording test.
  • No e2e scenario catalog update: internal/virtual is an internal library with no user-reachable surface until the virtual repository HTTP handlers (S30–S32) compose it.

Related to #333 (closed)

:bricks: Stacked MRs (review/merge bottom-up)

Edited by David Fernandez

Merge request reports