Record River's accepted database metrics surface in S27, and request a LabKit QueryTracer option

Split from #1606 (closed) (gap 2, part 1 of 2). Parent epic: &24088. Part 2, the counters-only River tracer, is #1722.

Decision (operator ruling 2026-10-25)

Q4: ship the decision record now, ask LabKit for a tracer option, and then add a counters-only River tracer when LabKit ships the option (#1722). This issue is the decision record and the LabKit request. It is docs-only, and it is ready now.

Problem

River runs over the application's *sql.DB and shares its pool. It polls every 500 ms per pod and runs up to 25 workers. None of its queries carries a metric: JobGetAvailable, JobSetStateIfRunningMany, JobInsertFastMany, leader election, the rescuer and the cleaner. They show up only as pool acquires, OTel spans and CloudSQL totals. When the pool saturates, no metric tells River load apart from request load. #1664 states this caveat on PoolSaturated. S27 does not say what the accepted River database surface is.

Evidence

Research base: origin/main at e2c420456, 2026-10-25.

  • River shares the pool: internal/jobsriver/client.go:203 (river.NewClient(riverdatabasesql.New(c.db), cfg)) and cmd/artifact-registry/wire_jobs.go:196-205. The constants are at internal/jobsriver/client.go:36 (defaultMaxWorkers = 25) and :40 (defaultFetchPollInterval = 500 * time.Millisecond).
  • S27 justifies the shared pool with "One pool, one set of connection limits, one set of metrics" (docs/specs/S27-background-jobs-foundation.md:186-187). It claims tracing is inherited through the pgx QueryTracer (:194-210). No metric attributes pool use to River.
  • LabKit has no seam for a second tracer. postgres.NewWithConfig sets connCfg.Tracer = &queryTracer{...} (labkit v2@v2.63.2/postgres/client.go:207-209), and Start copies that private connCfg into the pool (:271). NewOption offers only WithInfraConfig, WithTracer and WithConnectionPool (:324-349). This is the same at the latest tag, v2.65.1. pgx v5.11.0 ships multitracer, so chaining is a small upstream change.
  • An application-side wrapper does not avoid the upstream change. A second *sql.DB over a wrapped stdlib connector adds its own database/sql connection layer on top of the pgx pool. It also misses InsertTx on a caller's transaction (cmd/artifact-registry/bulk_enqueuer.go:187), because that runs on the application's *sql.Tx. That option was rejected.

Fix direction

  1. Amend S27 #### Database connection and docs/dev/observability.md to record that pool stats, OTel spans and jobs_in_flight{backend="river"} (docs/dev/observability.md:71) are the accepted River database surface today.
  2. Say what that surface cannot show: River's share of pool time and of database time. Point to #1722 as the follow-up that adds per-query counters once LabKit can chain a tracer.
  3. File the LabKit feature request below in the LabKit project (operator action; the text is ready), and link it from this issue and from #1722.

LabKit feature request (draft)

Title: postgres: let callers chain an additional pgx.QueryTracer alongside the OTel tracer

postgres.NewWithConfig sets connCfg.Tracer = &queryTracer{...} (v2/postgres/client.go:207-209 at v2.63.2, unchanged at v2.65.1), and Start copies the private connCfg into the pool (:271). NewOption offers WithInfraConfig, WithTracer and WithConnectionPool only (:324-349), so a service cannot add its own pgx.QueryTracer without losing LabKit's.

Ask: a postgres.WithQueryTracer(pgx.QueryTracer) option (repeatable, or variadic) that LabKit chains with its own tracer through github.com/jackc/pgx/v5/multitracer. LabKit's tracer stays first and is always present.

Use case: Artifact Registry runs River over the shared pool and wants to count River's own statements (which start with -- name: <Query>) by query, without re-timing statements the application already times by name.

Size: about 50-100 lines plus a test that asserts both tracers see TraceQueryStart / TraceQueryEnd.

Acceptance criteria

  • S27 #### Database connection and docs/dev/observability.md record that pool stats, OTel spans and jobs_in_flight{backend="river"} are the River database surface, and name what they cannot show.
  • The docs point to #1722 for River query counters. They describe the current state only, with no promise of a date.
  • The LabKit feature request is filed and linked from this issue and #1722 (operator action, outside the smurfit unit). Covered by gitlab-org/labkit#154.
  • S27 stays Status: Approved.

Size and fit

About 30-60 lines of docs. This fits one smurfit unit.

Dependencies

None. #1664 does not need this issue. It documents the "may be job load" caveat on PoolSaturated.

Edited by Pawel Rozlach