chore(accounting): derive the detached-write caps from their resources (#559 stack: 1/2)
Summary
Problem. Every detached-write in-flight cap was a hand-rolled per-consumer constant of 64. Each one was sized alone against a resource that the rest of the process draws on too. The joint draw grows with every consumer, and nothing observes the total.
Impact. The service under load, and its operators: the joint draw could overdraw the Postgres connection pool and the cache-purpose Redis pool. Users saw no response change.
Change. The composition root now derives every in-scope ceiling from the capacity of the resource it draws on. The rule is max(floor_i, floor(C / (2N))), with N = 8 for the Postgres partition and N = 5 for the Redis partition. The floors are 6 for the two sub-quota parents (maven buffered update and npm counter emit) and 1 elsewhere. The Postgres capacity is Pool().Stat().MaxConns(). The Redis capacity is the cache client's resolved PoolSize. The documentation that tabled the removed constants now states the derived ceilings.
One provider per consumer package re-derives the ceilings on every read, and one setter per package installs it before any tier builds a handler. A cap's semaphore is made at its first acquire from the installed provider. A package with no installed provider refuses with a sentinel error instead of using a default. The sub-quotas below these ceilings (maven's per-label quotas, npm's per-site and per-package quotas, and OCI's per-repository quotas) derive from the same parents. Operator ruling q163 (option D) supersedes the capacity-and-sharing clauses of acceptance item A9 in the coverage table below. The OCI sink's stampRepository and emitDownload re-resolve the cap when the sink cached none. The OCI counter sink caches no cap at construction, because the composition root builds it before the Postgres pool opens. Its first dispatch sizes the counter-emit, repository-stamp and per-repository download caps.
Each budget read logs one Info line with consumer_package, each ceiling that the read returns, postgres_max_conns and redis_pool_size, so the log of a pod shows the ceilings that it uses.
Downsides. The reserve fraction (1/2) and the member counts (8 and 5) are code constants, not configuration. The per-package materializer block and the splitQuota helper stay duplicated across packages, and issue #1701 (closed) tracks the deduplication.
Checklist
- Ran it: Cold
driver.sh smokefrom therun-artifact-registryskill: a full rebuild, a boot against real Postgres, MinIO and Redis, and 103 route checks over HTTP with curl. Result:smoke: 103 passed, 0 failed. That run is atf19e0e15a, a commit from before the branch was squashed, and no commit on this branch has its tree.0fe404950contains that run's tree plus sixteen later review-fix commits, and the smoke check did not run again on0fe404950or any later commit. No package-manager client ran: the format conformance suites could not start locally, and their CI jobs are the evidence. - Interfaces and data: None. The change adds no API, configuration key, schema, or wire or storage format change. The ceilings read the Postgres pool size and the cache client's resolved pool size, which the configuration already sets. No spec governs this issue, and no
docs/user/page changes, because no customer-visible surface moves. - Failure: An unwired package refuses: the dispatch does not run its work, the package logs the sentinel error, and no shed is metered. A cache client whose pool size cannot be read fails boot. In a wired process both paths are unreachable, because the budgets are installed before any tier builds a handler, so the unit tests drive them and a client sees no change. Shedding is unchanged: a saturated site still drops without blocking and increments its dropped series.
- Rollout: n/a. The change adds no schema and no configuration key, so a previous release runs unchanged against the same database and configuration during a deploy.
- Hot path: No query and no unbounded work. Each dispatch takes a short per-package mutex and reads its already-materialized semaphore. The provider runs at a cap's first acquire only, and it reads the pool's configured maximum connections, which issues no SQL.
- Database: No. No migration and no query changes. The one new database read is the pool's configured maximum connection count (
Pool().Stat().MaxConns()), which issues no SQL. - Security: No. The change adds no input, route, permission or audited action, and it does not change how callers are authenticated or authorized.
- ADRs: None cited. The issue body and the card name no ADR, and the validation pass found no ADR conflict.
- E2E: No scenario row is added or changed, and no file under
docs/testing/is in the diff. The change adds no endpoint, RPC, migration, storage path, configuration knob or middleware, so no catalog row observes it. The card type ischore, notfeatorfix. - Simplicity: The issue rules out two simpler versions: halving the caps (still a per-consumer guess) and one process-wide semaphore shared between two packages (a shared-instance seam neither package owns). A single
func() intprovider was rejected too: one int cannot carry the several caps that maven, npm, OCI and managementapi each hold. Each package gets one provider that returns one ceiling per cap.
Coverage
No spec governs this issue; the rows are the card's derived acceptance items (A1-A12 with A3b, 00-setup-enrichment.md "## Enriched card"), each with the artifact it was read from.
Acceptance items
| # | Item | Tests |
|---|---|---|
| A1 | Each ceiling is max(floor_i, floor(f x C / N_C)): f = 1/2, N_P = 8, N_R = 5, floors 6 (maven buffered update, npm counter emit) and 1 elsewhere | TestDeriveInFlightBudgets |
| A2 | A partition's ceilings sum to at most f x C, and a further member does not raise that sum | TestDeriveInFlightBudgets_PartitionSumsStayInsideTheReserve, TestInFlightShare_AddingAMemberNeverRaisesThePartitionSum |
| A3 | P is client.Pool().Stat().MaxConns(), read at first use and not at composition | TestAppPoolMaxConns_FixesNoCeilingBeforeStart, TestCounterSink_BuiltBeforeThePoolOpensTakesTheCeilingAtTheFirstDispatch (oci) |
| A3b | R is the live cache client's resolved PoolSize by client type; any other type is an explicit error, never a default | TestCachePoolSize_ReadsTheResolvedPoolSizeByClientType, TestWireInFlightBudgets_RefusesAClientWithNoPoolSize |
| A4 | One startup setter per package; every in-scope channel is made once, at its first acquire, from that provider | TestBufferedUpdate_TakesItsCeilingFromTheInstalledBudget (maven, npm), TestCounterEmit_TakesItsCeilingFromTheInstalledBudget (maven, npm), TestRebuild_TakesItsCeilingAndPerPackageSubQuotaFromTheInstalledBudget, TestCounterEmit_TakesItsCeilingsFromTheInstalledBudget (oci), TestDownloadSignals_TakeTheirCeilingsFromTheInstalledBudget, TestEmitChunkCounters_TakesItsCeilingFromTheInstalledBudget, TestBufferedCounterUpdate_TakesItsCeilingFromTheInstalledBudget, TestDeleteCounterEmit_TakesItsCeilingFromTheInstalledBudget, TestRecordDownload_ResolvesTheDerivedCapAtTheFirstAcquire |
| A5 | An unwired package refuses with an error: the work does not run and no default ceiling applies | TestBufferedUpdate_UnwiredPackageRefusesWithAnError (maven, npm), TestCounterEmit_UnwiredPackageRefusesWithAnError (maven, npm, oci), TestRebuild_UnwiredPackageRefusesWithAnError, TestEnqueueRebuild_UnwiredPackageRefusesTheRebuild (npm), TestCounterSink_UnwiredPackageRefusesTheDownloadAndTheStamp (oci), TestDownloadSignals_Record_UnwiredPackageRefusesWithAnError (oci), TestEmitChunkCounters_UnwiredPackageRefusesWithAnError, TestBufferedCounterUpdate_UnwiredPackageRefusesWithAnError, TestDeleteCounterEmit_UnwiredPackageRefusesWithAnError, TestRecordDownload_UnwiredPackageRefusesWithAnError |
| A6 | Every cap still sheds without blocking, and a shed still increments the site's dropped series | TestCounterEmit_TakesItsCeilingFromTheInstalledBudget (maven, npm), TestEmitChunkCounters_TakesItsCeilingFromTheInstalledBudget, TestDeleteCounterEmit_TakesItsCeilingFromTheInstalledBudget, beside the pre-existing shed suites the card lists |
| A7 | The Postgres caps and the Redis caps stay separate semaphores | TestCounterEmit_SaturatingTheRedisArmShedsNoPostgresArmDispatch |
| A8 | inlineBuildMaxInFlight stays const 64 and S11-npm-hosted.md:2726 is unchanged | No Go test: the constant is not on any budget path and the row is a do-not-change, carried by the implementer (see gaps) |
| A9 | The composition root calls every setter; each channel has its derived capacity after first use; two constructions share one channel | TestWireInFlightBudgets_InstallsEveryConsumerSetter, TestInFlightBudgetProviders_InstallReachesEverySetter, TestNewInFlightBudgetProviders_ReadPostgresAtEveryBudgetRead, TestDownloadSignals_TakeTheirCeilingsFromTheInstalledBudget, TestRecordDownload_ResolvesTheDerivedCapAtTheFirstAcquire |
| A10 | The per-label and per-site quotas sum to their parent at every parent of 6 or more with no member at 0; per-key ceilings are 1/16 and 1/4 of the parent, minimum 1 | TestBufferedUpdate_PerLabelQuotasSumToTheDerivedParent, TestCounterEmit_PerSiteQuotasSumToTheDerivedParent, TestRebuild_TakesItsCeilingAndPerPackageSubQuotaFromTheInstalledBudget, TestCounterEmit_TakesItsCeilingsFromTheInstalledBudget (oci) |
| A11 | The census command returns exactly the caps its tables list | Owned by the documentation change in docs/dev/storage-accounting.md; no Go test (see gaps) |
| A12 | No sentence still states each consumer admits its own 64 | Owned by the documentation changes in docs/dev/storage-accounting.md, docs/dev/remote-read-budgets.md and docs/dev/observability.md; no Go test (see gaps) |
Gaps
-
A8, A11 and A12 are documentation and do-not-change rows with no Go test here; the implementer's doc change and its untouched constant carry them.
-
A8:
inlineBuildMaxInFlightis stillconst 64atinternal/format/npm/inline_build.go:27, and nodocs/specs/file is in the diff, sodocs/specs/S11-npm-hosted.md:2726is unchanged. -
A11: the census and its command live in
docs/dev/storage-accounting.md, under the section "In-flight caps of this shape, by the resource they draw on". No Go test asserts its count, and the check is that command,git grep -nE "^func [A-Za-z]+Semaphore\(\) \(chan struct\{\}, " -- internal, which the validation pass ran by hand. -
A12: the three pages the row names (
docs/dev/storage-accounting.md,docs/dev/remote-read-budgets.mdanddocs/dev/observability.md) are updated on this branch, and no Go test pins prose, so the pages are the evidence.
The other two gaps in that list are closed on this branch: the suites that pinned a literal 64, and the test setups that needed a budget installed, both landed in 0fe404950.
Size
Size at head c665f83653a93a8d2cce3381d663948ef7b0e419: 4,359 reviewable LOC across 77 files — Go production 1,661 across 26 files, Go tests 2,378 across 41 files, documentation 320 across 10 files (added Go lines, code and tests together, 3,603).
Deferred to a separate merge request
The run recipe, .claude/skills/run-artifact-registry/SKILL.md, still describes the purger's counter-emit cap as a fixed 64-slot semaphore. This branch removes that constant, so that section is stale. A guardrail file changes in its own merge request, so the correction is drafted and left to a separate guardrail merge request, opened by hand outside this unit.
Derived ceilings
Each ceiling is a formula in two inputs, P and R.
- P is
MaxConnsof the Postgres pool. It ispostgresql.max_connswhen the configuration sets it. Else pgxpool usesmax(4, NumCPU). - R is
PoolSizeof the cache-purpose Redis client. It isredis.cache.pool_sizewhen the configuration sets it, else the infrastructurepool_size. Else go-redis uses10 * GOMAXPROCS.
The Postgres share is floor(P / 16), and the Redis share is floor(R / 10).
The last column gives the value at R = 20, the go-redis default for this deployment: the 500m CPU limit rounds up to 1, and the Go runtime sets GOMAXPROCS to 2 or more.
| Cap | origin/main |
Formula | Value at R = 20 |
|---|---|---|---|
maven BufferedUpdate |
64 | max(6, floor(P / 16)) |
P is unknown |
maven CounterEmit |
64 | max(1, floor(R / 10)) |
2 |
npm BufferedUpdate |
64 | max(1, floor(P / 16)) |
P is unknown |
npm Rebuild |
64 | max(1, floor(P / 16)) |
P is unknown |
npm CounterEmit |
64 | max(6, floor(R / 10)) |
6 |
oci CounterEmit |
64 | max(1, min(floor(P / 16), floor(R / 10))) |
1 or 2 (P is unknown) |
oci RepositoryStamp |
64 | max(1, floor(P / 16)) |
P is unknown |
oci DownloadSignal |
64 | max(1, floor(P / 16)) |
P is unknown |
lifecycle CounterEmit |
64 | max(1, floor(R / 10)) |
2 |
managementapi BufferedCounterUpdate |
64 | max(1, floor(P / 16)) |
P is unknown |
managementapi DeleteCounterEmit |
64 | max(1, floor(R / 10)) |
2 |
remote DetachedBump |
64 | max(1, floor(P / 16)) |
P is unknown |
origin/main has 13 caps at 64, and this table has 12 rows.
The 13th cap, npm inlineBuildMaxInFlight, stays 64 and is not derived (acceptance item A8).
The deployed value of P is not known, because the repository does not contain it.
Each pod publishes its P on the gauge gitlab_artifact_registry_database_connection_pool_max_size.
R = 20 is correct only if ar-config sets no redis.cache.pool_size and the infrastructure configuration sets no pool_size.
Related to #559 What remains: the issue states no outcome of its own. Its acceptance items were derived by the enricher, not stated by the issue author.
This is a bot message