chore(virtual): add the merge budget's metrics and saturation alert
Summary
- Problem.
virtual.MergeBudget(internal/virtual/merge_budget.go) caps the bytes that the Maven and npm virtual metadata merges hold at once. The budget took and released its holds throughsemaphore.Weightedand recorded nothing, and the composition root never registered a collector for it. A budget that was always full read exactly like an idle one. - Impact. An operator cannot see how full the budget is, how long a read holds it, or how long a read waits for it. A saturated budget parks merging reads and burns no CPU, so no other signal moves.
- Change. The budget now records four label-free families, served on
/-/metricsasgitlab_artifact_registry_virtual_merge_budget_in_use_bytes,gitlab_artifact_registry_virtual_merge_budget_capacity_bytes,gitlab_artifact_registry_virtual_merge_budget_hold_duration_seconds, andgitlab_artifact_registry_virtual_merge_budget_wait_duration_seconds.NewMergeBudgetsets the capacity gauge.MergeBudget.holdraises the in-use gauge and its release lowers it, observing the hold once.AdmitandTryAdmitobserve the wait around the acquisition.cmd/artifact-registryregisters the families at boot, besidestorage.RegisterMetrics.docs/dev/observability.mdcarries one row per family, anddocs/dev/alerting.mdthe saturation rule on the in-use-to-capacity ratio. - Downsides. Each admitted merge takes two more clock readings and two more histogram observations, plus a gauge add and subtract. Both histograms first shared
metrics.DBLatencyBucketswith the database-query family. The review's fix commit 62d80188 replaced them with the package-localmergeBudgetDurationBuckets(1ms to 600s). The wait histogram books a near-zero sample for aTryAdmitthat takes its hold, because that path never parks. The alerting rule itself lands in the external alerting file thatdocs/dev/alerting.mdnames.
Checklist
- Ran it: The runtime pass booted this branch's binary (
driver.sh up,seed,smoke103/103) and read/-/metricswithcurl, at heade5a654fd3. All four families were present and zero before any read, withcapacity_bytesat the boot-derived1.11149056e+08. A hand-built virtual Maven merge (a virtual repository over the seeded hosted one, thenGETof the mergedmaven-metadata.xml→ 200) moved them as designed: both histograms0→1, andin_use_bytesback to0. The npm route cannot be driven live, because it refuses before the budget code runs without a fixedNPM_PUBLIC_REGISTRY_URL, and the harness refuses theVAR=valueboot form. Maven and npm share one*virtual.MergeBudget, so the live reading is Maven's. The parked and shed waits need two overlapping requests, which one session cannot make. The unit tests cover both. - Interfaces and data: It adds no API, configuration, schema, wire, or storage format change. The new surface is four metric families on the existing
/-/metricsendpoint, under the existinggitlab_artifact_registry_prefix. Each family is label-free, so existing scrapers keep working. S13 states no metric contract for this surface, so the spec needs no change.docs/user/documents no metric surface, so no user page moves. The two operator pages the issue names carry the new rows and the saturation rule. - Failure: The change adds no new failure mode for a read. If the registration of a family fails, the composition root panics at boot with
composition: registering virtual metrics: ..., so the process fails fast instead of serving reads that nothing scrapes. A client sees no change: a read either succeeds, or it parks until a hold frees or its wait bound passes. - Rollout: n/a. The change touches no schema and no configuration, so the previous release runs unchanged. The new series appear only on the pods that carry this change, and a mixed fleet scrapes both.
- Hot path: It adds two clock readings and two histogram observations to every admitted merge, plus a gauge add and subtract. All of them are constant work on the budget's existing admit and release path, which the merge already runs. The families are label-free, so the series count is fixed. No query, allocation, or unbounded work is added.
- Database: n/a — no migration and no query change.
- Security: n/a — it adds no input, route, permission, or audited action, and it changes no authentication or authorization path. The families carry no labels, so the endpoint exposes no namespace, repository, or user identity.
- ADRs: None governs this surface. No ADR in
docs/adr/covers the budget's metrics or the alerting page. The change keeps the existing structure: the collectors live in the package that owns the budget, and the composition root registers them once at boot. - E2E: None. The change adds no handler, route, status code, or response body, so no client-observable journey changes and
docs/testing/needs no row. - Simplicity: The issue offered two forms for the wait reading: a histogram, or a
budget_wait_sfield on thenpm_virtual_resolveandmaven_virtual_resolveevents. The event-field form is rejected, because both formats share one budget and the issue's scope sentence puts the metrics onvirtual.MergeBudget, not in a format package. Inlining the registration call intoregisterServiceMetricswas also rejected, because the function then passesfunlen's statement limit.
Coverage
Source: issue #1680 (closed)'s card acceptance (the card's spec field names S13,
whose "Merge memory budget" section states no metric contract for this surface;
its two budget criteria are listed below).
| # | Acceptance item (card) | Tests |
|---|---|---|
| A1 | A dashboard can show budget use against capacity, and hold and wait times, for Maven and npm reads. | TestMergeBudget_MetricsTrackInUseAgainstCapacity (capacity and in-use), TestMergeBudget_MetricsObserveTheHoldAtRelease (hold at release, repeated release, and the no-observation case a leak rests on), TestMergeBudget_MetricsObserveWaitDurations (free and parked waits), TestMergeBudget_MetricsObserveTheWaitOfAnEndedAdmission (the shed wait, no hold), TestMergeBudget_MetricsFollowTryAdmit (the TryAdmit fast path npm takes first), TestVirtualCollectorsAreLabelFree (one shared series for both formats), TestWiring_VirtualMergeBudgetFamiliesRegistered (the app serves them) |
| A2 | docs/dev/observability.md lists every new metric. |
Not a Go test: the four rows are the implementation author's edit. TestRegisterMetrics_ExposesAllVectors pins the four family names a row must carry, and TestWiring_VirtualMergeBudgetFamiliesRegistered pins that the app serves them under the service prefix. |
| A3 | docs/dev/alerting.md has a saturation entry for the budget. |
Not a Go test: the entry is the implementation author's edit. TestMergeBudget_MetricsTrackInUseAgainstCapacity pins the in-use and capacity pair the ratio rule reads, and TestMergeBudget_MetricsObserveTheHoldAtRelease the short-holds-versus-leak reading beside it. |
S13 criteria this change rests on
| # | Spec row | Tests |
|---|---|---|
| S13 acceptance, "The merge memory budget admits a contiguous prefix" | Admission semantics are unchanged; this change only records them. Existing TestMergeBudget_Admit, TestMergeBudget_AdmitNeverExceedsTheBudget, TestMergeBudget_AdmissiblePrefix cover it. |
|
| S13 acceptance, "Admission parks until the budget frees" | Unchanged. Existing TestMergeBudget_AdmitParksWhileTheBudgetIsHeld, TestMergeBudget_AdmitReturnsTheContextError, TestMergeBudget_ReleaseFreesTheBudget cover it, and the new wait cases record the same park. |
Every other S13 acceptance criterion, error case and security consideration is outside this change: it adds no behavior to the resolver, the cache or the upstream client, and the spec states no metric contract for the budget.
Merge order
!3470 inserts a new top-level function at the same anchor in cmd/artifact-registry/main.go that this branch uses: after registerServiceMetrics, before appLogConfig. The collision is recorded and not resolved on this branch. Whoever merges second keeps both functions at the rebase.
!3460 also edits docs/dev/observability.md, and its change conflicts with this branch in that file. The conflict is recorded and not resolved on this branch. Whichever MR merges second resolves it at its rebase.
Closes #1680 (closed)
This is a bot message