feat(maven): wrap metadata reconciler in a River background job

Moaz Khalifarequested to merge
mkhalifa/maven-hosted-step-10 into main

What

Step 10 of the S10 Maven hosted plan: the async layer over the Step 9 MetadataReconciler. Wraps Reconcile in a River background job so a version soft-delete (S20, forthcoming) can enqueue package-level maven-metadata.xml reconciliation that survives a process crash and dedupes per Maven package.

Ships the maven-package seam — four symbols in internal/format/maven/reconciler_job.go:

  • ReconcileArgs{NamespaceID, RepositoryID, PackageID uuid.UUID} — the job payload; Kind() == "MavenMetadataReconciler" (stable routing / dedup key).
  • reconcilerWorker — river.Worker[ReconcileArgs]; delegates to Reconcile, recovers a panic (logs args / panic / stack via the request-scoped logger, then cancels) so one poisoned package cannot crash the worker pool.
  • EnqueueReconcile(ctx, *jobsriver.Client, *sql.Tx, ReconcileArgs) — the enqueue helper S20 calls inside its soft-delete transaction; atomic with the caller's *sql.Tx, deduped via WithUnique(UniqueByArgs()).
  • RegisterMavenReconcilerJob(*jobsriver.Client, MetadataReconciler) — worker registration for the composition root.

Deferred to S20 (surfaced, not built here)

The plan's Files: for Step 10 listed cmd/artifact-registry/main.go (register the worker) and docs/dev/background-jobs.md. Both are deferred:

  • Maven is not production-mounted until Step 10b (!916 (merged), open), whose buildMavenDispatcher constructs only the HTTP dispatcher — no maven.Reconciler.
  • wire.go calls wireJobs (where RegisterWorker must run, before River.Start) before the Maven store / BlobStore dependencies exist, so registering the worker needs composition-root restructuring — and there is no enqueuer (S20) yet. Wiring it now would be speculative plumbing that conflicts with the open Step 10b.

RegisterMavenReconcilerJob is ready for S20 to call; S20 owns resolving the registration ordering and the background-jobs.md entry.

Decisions

  • depguard / *sql.Tx. EnqueueReconcile's plan-mandated *sql.Tx (required by jobsriver.EnqueueTx, atomic with the caller's transaction per S27 AC #5) forces a database/sql import that ADR-023's no-direct-db rule blocks in format packages. Resolved with a file-scoped .golangci.yaml exception — matching this repo's convention of file-globbed depguard rules (as with the auth-bootstrap exemption on main.go), not an inline nolint. The import threads a transaction handle to the jobs framework; it issues no SQL.
  • Worker sentinel → River result mapping. nil / ErrNoMetadata → complete (nothing to reconcile is a benign no-op); ErrInvalidMetadata / ErrEmptyVersionSet → river.JobCancel (terminal — a DOCTYPE rejection is deterministic and an empty set will not grow on retry); anything else → returned so River retries. The ErrEmptyVersionSet orphan-row cleanup stays S20's soft-delete-cascade concern; the worker holds only a MetadataReconciler (no store) by design.

Testing

3 unit tests + 7 integration tests (River backend via testcontainers). The full Maven integration suite passes locally (go test -tags integration ./internal/format/maven/..., green), and the Step 10 job tests run against a real River/PostgreSQL backend.

## Spec coverage

Spec: docs/specs/S10-maven-hosted.md (Async work; Background-job amplification; Reconciliation on deletion)
Plan: docs/plans/2026-05-12-maven-hosted.md (Step 10)

Step 10 wraps the Step 9 MetadataReconciler in a background job. It owns the
async delivery contract (enqueue, dedup, idempotent execution, panic
resilience, drain); the reconcile logic and its error sentinels are Step 9's,
verified in reconciler_test.go. Tests marked (integration) require
`-tags integration` + a live PostgreSQL (testcontainers).

### Acceptance criteria

| #      | Criterion                                                             | Tests                                                                                                                                                                                 |
|--------|-----------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| AC 31  | `Reconcile` after `SoftDeleteVersion` rewrites metadata to live set   | Rewrite logic owned by Step 9 (`TestReconciler_Reconcile_RewritesDroppedVersion`); async delivery to `Reconcile` with exact coordinates re-verified by `TestReconcileJob_EnqueueAndProcess` (integration) |
| AC 32  | No-op fast path on coherent metadata writes no blob                   | Step 9 (`TestReconciler_Reconcile_CoherentMetadataNoOp`); re-verified via the async path (duplicate delivery, zero blob write, zero row mutation) by `TestReconcileJob_IdempotentNoOpOnCoherentState` (integration) |
| AC 33  | No metadata row returns `ErrNoMetadata`, no row creation              | Owned by Step 9 (`TestReconciler_Reconcile_NoMetadataRow`); worker-level sentinel handling is a reconcile-service concern, not asserted here                                          |
| AC 1–30| Download / upload / sidecar / path-traversal / envelope behavior      | Owned by Steps 5–8b; not in Step 10 scope                                                                                                                                             |

### Async-work contract (spec "Async work")

| Property                                                                                   | Tests                                                                                              |
|--------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------|
| Stable job kind `MavenMetadataReconciler` (durable routing/dedup key)                      | `TestReconcileArgs_Kind`                                                                            |
| Worker unmarshals args, delegates to `Reconcile` with exact coords; transient errors retry | `TestReconcilerWorker_DelegatesToReconciler`; `TestReconcileJob_EnqueueAndProcess` (integration)   |
| Enqueue atomic with the caller's `*sql.Tx` (rolled-back tx enqueues nothing)               | `TestReconcileJob_EnqueueRollbackNotProcessed` (integration); `EnqueueTx` atomicity is jobsriver-owned |
| Idempotent execution under at-least-once delivery (no-op fast path on re-run)              | `TestReconcileJob_IdempotentNoOpOnCoherentState` (integration)                                     |
| Panic in one reconcile recovered + logged (args, panic, stack); pool survives              | `TestReconcilerWorker_PanicRecovered`; `TestReconcileJob_PanicDoesNotKillPool` (integration)       |
| In-flight reconcile drains within the framework deadline on shutdown                       | `TestReconcileJob_DrainCompletesInFlight` (integration)                                             |

### Security considerations

| Concern                                                                                       | Tests                                                                                                                             |
|-----------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
| Background-job amplification (T-06): per-package enqueue dedup bounds worker-pool amplification | `TestReconcileJob_PerPackageDedup` (integration) — two enqueues for the same package collapse to one; a distinct package is not deduped |
| Background-job amplification (T-06): `max_metadata_size` (10 MB) + `max_versions_per_package` (25,000) | Enforced in the reconcile service (Step 9): `TestReconciler_Reconcile_OversizedMetadataRejected`, `TestReconciler_Reconcile_TooManyLiveVersionsRejected` |
| `maven-metadata.xml` DOCTYPE / billion-laughs rejection on the reconcile parse path            | Owned by Step 9 (`TestReconciler_Reconcile_DoctypeRejected`)                                                                       |
| Path traversal, coordinate validation, `detail` hygiene, 404 info disclosure, body size, slowloris, sidecar bounds, FIPS MD5, delivery-mode dispatch, storage tenancy, audit trail | Owned by Steps 5–9; not in Step 10 scope |

### Error cases

| Condition                                                                          | Tests                                                                        |
|------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
| Enqueue when the backend is unavailable → error surfaced to caller                 | jobsriver-owned (`ErrBackendUnavailable`); inherited by `EnqueueReconcile`   |
| Handler error cases (400/401/403/404/405/409/413/416/422/429/500 envelopes)        | Owned by Steps 5–8b; not in Step 10 scope                                    |

Related to #21 (closed)

Merge request reports