Grouped follow-ups from the S22 storage-accounting spec ([!1121](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1121)) that can only be settled against real production metrics, not pre-sized. Revisit each once the S22 pipeline is running and `reconciliation_backlog`, `reconciliation_drift_bytes`/`reconciliation_drift_count`, `counter_drain_chunk_bailed_total`, and `jobs_duration_seconds{kind=...}` carry real data. - **Reconciliation throughput at scale.** Confirm the per-namespace task cost (`repo_count + 1` sequential Redis clears/scans/write-backs) and the standing schedule drain the stale backlog within `reconciliation_interval`. Levers if it falls behind: `reconciliation_max_in_flight`, `reconciliation_interval`, read-replica routing, partition-aligned fan-out. - **Drain reconciliation-guard overhead.** Verify the per-chunk baseline `SELECT` of `last_reconciled_at` stays negligible at real drain volume; revisit folding it into the merge step or dropping it for the namespace-scoped family. - **Drain-vs-drain single-flight closure.** The stale-claim admission check narrows the residual double-count and three windows survive it: a chunk admitted fresh that then executes slowly under database contention, a trigger double-fire, and a fan-out commit whose outcome the trigger never learned. Revisit a per-scope single-flight mechanism only if `counter_drain_chunk_bailed_total` together with `reconciliation_drift_*` show it is material. - **Partition-aligned reconciliation fan-out.** Revisit only if metrics show per-task fan-out (not per-query cost) is a bottleneck the catch-up loop and `reconciliation_backlog` do not already contain. - **Finer `reconciliation_backlog` refresh resolution.** If once-per-`reconciliation_interval` sampling proves too coarse for alerting, move to a scrape-time leader-side collector or a short-interval leader-elected sampler (keep a single writer). Deferred from S22; see the spec's Follow-ups section. ## Two more, folded in from work item 934 Both are measurements rather than design questions, so neither can be settled before the pipeline carries real load. That is the same condition every bullet above carries, which is why they are here rather than in an issue of their own. !1973 converts the S22 spec into a landing stub, so its Follow-ups section stops being their record. The measurements and the design notes survive the conversion, in `docs/dev/storage-accounting.md`. What does not survive is the statement that somebody has to check either number once there is production traffic. This section is that statement. ## 1. Confirm the orphan sweep's candidate page bound against production cardinality The orphan sweep's page is an anti-join whose `LIMIT` does not stop its two sequential scans. `docs/dev/storage-accounting.md` records the measurement: roughly 1,456 buffers and 11.7 ms at 50,000 namespaces. The 5 s bound on one page read is argued from the same operational budget that bounds the `namespaces_missing_statistics` count, which leaves roughly 430x of headroom against that measurement. Both figures behind that multiple are warm reads. The open question is the cardinality at which a cold or contended page reaches 5 s. That depends on the deployed row count and on storage latency, and neither the spec nor the document pre-sizes it. **How to settle it:** read the sweep's own `jobs_duration_seconds{kind="trigger:reconciliation_orphan_sweep"}` against the deployed `namespaces` row count, on a cold cache rather than a warm one. ## 2. Confirm the per-scope merge round-trip cost against production data The merge-to-`:flushed` script runs once per scope, so one chunk is up to 850 sequential Redis round trips at the largest `drain_chunk_size` the configuration layer admits. Whether that shape needs batching is a measurement rather than a design question. **How to settle it:** read `gitlab_redis_command_duration_seconds{name="redis-cache"}`, which the cache-purpose client already exports and whose buckets separate under 1 ms from 1 to 5 ms from 5 to 25 ms. The arithmetic the document records: - At 5 ms per round trip a 500-scope chunk costs 2.5 s. - One tick's 80 chunks (`drain_batch_size` 20000 over `drain_chunk_size` 500, for each of the two families) cost about 200 chunk-seconds, which is 27% of the 750 worker-seconds a pod running 25 chunk workers offers across a 30-second `drain_interval`. - Crossing River's one-minute job budget needs about 71 ms per round trip at 850 scopes, or about 120 ms at 500. ### If the figures justify batching, the batched form needs its own `NOSCRIPT` strategy This is the part that is easy to get wrong, so it is recorded here rather than left to be rediscovered. go-redis' `Pipeliner` satisfies its `Scripter` interface, so composing the script call inside `Pipelined` compiles. The pipeline queues the command and returns a nil error, so the script call never sees the `NOSCRIPT` that triggers its `EVAL` retry. go-redis' own pipeline retry has no `NOSCRIPT` arm either. A `SCRIPT FLUSH`, a Redis restart, or a failover to a node that never cached the digest would then fail every chunk on every attempt. Three ways to close it: 1. A `SCRIPT LOAD` first. 1. An `EVAL` carrying the full script body per scope. 1. `NOSCRIPT` detection over the returned commands. ### What settles these two 1. Each figure is read off the metric named in its own section above, against production traffic, and the result is recorded on this issue. 1. If either exceeds its bound, this issue names the lever taken and links the change. 1. If both hold, the reader of this issue gets a measurement rather than an estimate. ## Read-replica routing is tracked apart, on purpose The first bullet above names read-replica routing as a lever, and https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/436 owns it. That issue is **not** folded here, and the difference is its trigger. Everything on this issue waits on a metric reading. Work item 436 waits on read replicas existing at all, which is an infrastructure event rather than a measurement. Folding it in would put it behind a gate it does not share, and a reader of this issue would wait for data that has no bearing on it. ## S22 alerting constraints, moved from #354 [#354](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/354) covers Grafana dashboards and alerting for the whole service, and it grew too large to carry per-sprint detail. The S22-specific constraints an alert author inherits are moved here. Each one is reproduced as a note on this issue, unchanged, and links back to where it was first posted. | Constraint | Note | | --- | --- | | `namespaces_missing_statistics` re-emits a cached value while its count keeps failing, so a zero does not prove the schema invariant holds, and Prometheus cannot tell the two apart | [note_3745070448](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/435#note_3745070448) | | The backlog collector is the first collector in this service that reads the database during a scrape, so its two 5 s budgets are a floor on the whole `/-/metrics` response | [note_3745070536](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/435#note_3745070536) | | The required `status="discard"` alert on `reconciliation:namespace` selects a state another sentence of the same spec forbids the handler from reaching | [note_3745070636](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/435#note_3745070636) | | No defined alert covers `trigger:counter_drain_repo` or `trigger:counter_drain_namespace`, and neither drain metric separates a stalled trigger from an idle fleet | [note_3745070795](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/435#note_3745070795) | | The discard alert cannot see a lease that expires on the final attempt, because asynq's recoverer archives that task without calling the error handler | [note_3745070892](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/435#note_3745070892) | | A reconciliation trigger fire discarded during a rolling deploy emits one `discard`, no `error`, and no `ok` | [note_3745071276](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/435#note_3745071276) | The three required paging alerts still land with #354, which owns the metrics-catalog entry and the rule files in `gitlab-com/runbooks`. What moves here is the S22 design detail those rules have to be written against. The two production-scale obligations this issue does **not** carry — the orphan sweep's candidate page bound and the per-scope merge round-trip cost — are tracked in [#934](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/934).