## What this tracks Three findings from the review of https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1916 answer one incident: a Redis that is slow, or refusing, while the lifecycle purger runs. They are cheaper to design together than to answer one at a time. 1. The purger's counter emit has no per-attempt bound. 1. Under a refusing Redis the drop log floods. 1. No metric counts issued or dropped deltas, so neither shape gives a clean signal. That merge request keeps the emit inline and records the decision in `emitChunkCounters`' doc. This issue carries what that decision leaves open. ## 1. The emit has no per-attempt bound `emitChunkCounters` (`internal/lifecycle/emit.go`) runs on the goroutine that just committed the chunk, from `internal/lifecycle/chunk.go`. While https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1916 is open the purger emits nothing. Once it merges, every committed chunk carries the wait below. `accounting.Emitter.record` discards the caller's deadline and builds its own, bounded by `detachedEmitTimeout` of 5 seconds (`internal/accounting/emit.go`, merged). A chunk can fire both arms, so one committed chunk can wait up to 10 seconds. The emit context is detached, so nothing interrupts that wait: not the attempt deadline, not River's graceful drain, and not River's force-cancel. `forceCancelGrace` is 5 seconds (`internal/jobsriver/client.go`), and one chunk's two emits can take twice that. What a stalled Redis then costs, from figures already in the tree: - An artifact-scope attempt scans seven levels at 100 tombstones each, so roughly 700 to 1,400 chunks, at the 20 to 100 ms per chunk the code prices (`internal/lifecycle/chunk.go`). That is 14 to 140 seconds of work. - At up to 10 seconds of emit per chunk, the 30-minute attempt (`purgeAttemptTimeout`, `internal/lifecycle/purge.go`) reaches about 180 chunks. - The attempt then runs out of clock, and `Work` returns the interruption as an error. `PurgeMaxAttempts` is 1, so River discards the job. The tombstone stands and the sweep re-queues the namespace. - One of the pod's 25 shared River workers (`internal/jobsriver/client.go`) is held for the full 30 minutes instead of seconds. - A force-cancelled attempt leaves its River row `running`. `UniqueByArgsWhileLive` counts that as live, so that `(namespace, scope)` pair is deduplicated out of the sweep for about an hour. ## 2. The drop log under a refusing Redis A refusing Redis fails at once, so the purge keeps its throughput and the log takes the load instead. `record` writes one Warn per dropped emit, carrying `counter_family` and `counter_scope` (`internal/accounting/emit.go`). An attempt of about 1,400 chunks with both arms firing writes about 2,800 Warn lines, per attempt, per namespace. ## 3. No metric for issued or dropped deltas Nothing counts the deltas the emitter issues or drops. A slow Redis and a refusing Redis therefore look the same in metrics, and a latency guard set too low is invisible. ## The shape proposed for the first part Time each emit at the call site. Once one emit passes a threshold, stop emitting for the rest of the attempt. That bounds an attempt's total emit cost at about one budget instead of one budget per chunk, so the purge keeps its throughput and the attempt does not fail on the clock. Two constraints on the design: - The call site cannot shorten the emit's own bound. `record` builds its own context and `Emitter` takes no budget, so a per-caller bound needs an `internal/accounting` API change. - The trigger has to be elapsed time. Neither emit method returns a value, so the call site cannot see a drop. Dropping the rest of an attempt's deltas is the posture the spec prescribes: `docs/specs/S20-a-lifecycle-closed-beta.md` says a purge must never fail because accounting was unavailable. Reconciliation recomputes both counters afterwards, at the 1-hour default of `storage_accounting.reconciliation_interval`. The guarantee behind that repair is the wired trigger in `cmd/artifact-registry/wire_accounting.go` and the unfiltered candidate walk in `ReconcileCandidateStore.ListStaleNamespaces`, which carries no serviceability predicate. ## Two alternatives already weighed, and why they are not this issue - **Dispatch behind a shedding cap**, the way the format packages do. River runs one queue at 25 workers per pod, across every job kind. A cap sized like its siblings at 64 slots can never engage, so the option is a spawn rather than a shed. It also pays a delta lost at process exit, and it adds a ninth copy of the shape https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/559 exists to unify. - **A shorter bound for this caller, or one derived from the attempt's remaining time.** This needs the `internal/accounting` API change named above. The accounting pipeline is out of scope for the S20-A spec, which owns the call sites alone. ## Related - https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/559 — unify the detached-write in-flight caps behind one pool-aware helper. - https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/767 — the same inline-emit question at the management API call sites, recorded as an unaddressed review thread from https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1754. - https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/632 — a shed counter write has no reconciliation of its own. - https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/435 — S22 drain and reconciliation follow-ups that need production metrics. _This is a bot message 🤖 — /smurfit_