docs(plans): plan the S27 queue-stats poller and settle its spec points

Summary

This plan builds the S27 Phase 2 area "Queue-stats poller and Redis coordination" for both job clients, River and asynq. It is the plan file docs/plans/2026-10-25-s27-queue-stats-poller.md. It covers the jobs_queue_depth and jobs_queue_oldest_age_seconds gauges, the jobs_queue_stats_ticks_skipped_total counter, the bsm/redislock lease that elects one pod per backend, and the metrics.* configuration block of both job backends. It also carries the S27 and S03-B spec corrections the plan depends on, following the S05-B plan MR's precedent. See the spec: S27, Queue-stats polling.

Two open issues read the gauges: the alert rules in #1663 and the dashboard in #1605. Until the gauges exist, a backlog that grows through rescued River jobs or lease-expired asynq tasks is not visible.

The plan costs ten step MRs. It leaves out the stuck-job rescue counter (#1692), the other Phase 2 areas, a kind label on the gauges, and the startup zero-initialization of the Phase 1 series (#1595 (closed)). Merging this MR approves the plan. No step MR opens before it lands.

Checklist

  • Risk first: Step 10, the step that wires the shared locker and turns the gauges on. It is the only step that changes what a running pod exports. Steps 2 to 9 merge inert. Steps 8 and 9 stop Step 10 from going first. The S27 sentence that says Phase 1 ships without the gauges waits for it.
  • Conflicts: The alert rules in #1663 and the dashboard in #1605 read the gauges, and S27 says this work starts after those two land. The operator reversed that order: this plan lands first, and this MR corrects the S27 text so that all three agree. The rescue counter in #1692 and the other Phase 2 areas stay outside this plan.
  • Reach: The alert rules in #1663 and the dashboard in #1605 read the gauges, so they need the series names, the backend label, and the seven-value state union before Step 10 turns the gauges on. The gauges do not exist yet, so the two readers get the series names from this plan and from Step 10. (draft, operator to confirm)
  • Coverage: No step delivers the stuck-job rescue counter (#1692), the other S27 Phase 2 areas, a kind label on the gauges, or the startup zero-initialization of the Phase 1 series (#1595 (closed)). The rescue counter and the zero-initialization are tracked in #1692 and #1595 (closed).
  • Least sure: Step 2, the lease holder, on the timing of the give-up rule. The lease_duration clock starts when the obtain or refresh is sent, not when the reply arrives. A measurement of the refresh round trip against the lease duration would change this answer. (draft, operator to confirm)

Approach

  • A new shared package, internal/jobs/queuestats, holds the lease holder, the tick loop, the gauge set, and the poller that composes them. Both clients import it. It imports neither River nor asynq.
  • accounting.Lease stays as it is. Its injection, Redis client, give-up rule, scrape-time reads, and component shape all differ from what S27 specifies.
  • Each backend registers its own gauge vectors, with backend as a constant label. So one backend's lease loss never removes the other backend's series.
  • A nil Options.Locker behaves like metrics.disabled: true. So Steps 2 to 9 merge inert, and Step 10 turns the gauges on in one place.
  • Shared machinery lands before its consumers, and each backend's reads land apart from its client lifecycle change. Measured predecessors (S27 Phase 1 Steps 4 and 7, S22 Step 16) were each about 2,000 added lines or more, because they bundled the two.

Steps

# Step Type Est. LOC Depends on
1 Queue-stats settings and startup validation feat ~330 none
2 Lease holder chore ~450 none
3 Queue gauges and tick-skipped counter chore ~250 none
4 Tick loop chore ~330 none
5 Poller composition chore ~450 1, 2, 3, 4
6 River queue-stats collector chore ~290 3
7 asynq queue-stats collector chore ~240 3
8 River client runs the poller chore ~340 5, 6
9 asynq client runs the poller chore ~310 5, 7
10 Wire the shared locker and document the series feat ~235 8, 9

Step MR titles end with (S27 queue-stats poller plan: <step>/10).

Operator rulings (2026-10-25)

  1. Lock keys follow the S05-A grammar with no hash tag: ar:lease:jobs:river_queue_stats and ar:lease:jobs:asynq_queue_stats.
  2. A leader gives up its lease after three failed refreshes, or once lease_duration has passed since its last successful obtain or refresh, whichever comes first. At the defaults that is about 60 s, not 90 s, so a leader cut off from Redis never exports past a lease that has expired.
  3. The job clients get a new lease holder. accounting.Lease is unchanged.
  4. The asynq tick calls Inspector.Queues() once, then GetQueueInfo for each listed tier, which is 1 + T calls. A tier nobody has enqueued to emits zeros, so an idle fleet still exposes the gauges.
  5. asynq.Inspector is used in production inside internal/jobsasynq only. It is built over the shared Redis client, never closed, and has no new lint rule.
  6. A nil Locker means no poller, the same as metrics.disabled: true.
  7. The tick-skipped counter registers at boot on the shared jobs.Metrics, not with the gauges, so a change of leader never resets it.
  8. River cancelled and River's own pending states are not emitted.
  9. #1693 (closed) lands first. The #1663 queue rules and the #1605 Queues row follow it.
  10. The cardinality audit's state pin widens to the union of the pool states and the queue states, seven values.
  11. The work ships as ten steps.

Spec corrections

  • docs/specs/S27-background-jobs-foundation.md:
    • Phases, Phase 2 list, and Queue-stats polling intro: #1693 (closed) ships first and unblocks #1663 and #1605.
    • What's polled (asynq): 1 + T calls, and zeros for an unlisted tier.
    • Coordination, Redis unavailable behavior, Error Cases, and the two lease_duration proto comments: the give-up rule, about 60 s at the defaults. The tick-body bullet and the refresh-alignment paragraph are removed because they repeat Tick lifecycle and Coordination. That keeps the spec, which is already over its word limit, from growing.
    • Lock keys and Out of Scope: the two S05-A keys, with no hash tag.
    • jobs_queue_depth state mapping: River cancelled and pending are dropped.
  • docs/specs/S03-b-metrics.md: the label budget table gains a state row for the S27 queue gauges.

Step 10 carries the one S27 change that must wait for the code: the sentence saying Phase 1 ships without the gauges.

Not affected

  • e2e: no new row. Only the lease holder exposes the series, and an e2e test cannot choose that pod through the ingress. The plan's Testing section gives the reasons.
  • docs/user/: no page covers metrics or the jobs configuration.
  • ADRs: ADR-006 and ADR-023 apply, and the plan conflicts with neither. The mirror's one unsynced upstream commit touches only ADR-004, 007, 009, and 010.

Related to #1693 (closed)

Edited by Pawel Rozlach

Merge request reports

Loading
Loading