Record River's accepted database metrics surface in S27, and request a LabKit QueryTracer option
Split from #1606 (closed) (gap 2, part 1 of 2). Parent epic: &24088. Part 2, the counters-only River tracer, is #1722.
Decision (operator ruling 2026-10-25)
Q4: ship the decision record now, ask LabKit for a tracer option, and then add a counters-only River tracer when LabKit ships the option (#1722). This issue is the decision record and the LabKit request. It is docs-only, and it is ready now.
Problem
River runs over the application's *sql.DB and shares its pool. It polls every 500 ms per pod and runs up to 25 workers. None of its queries carries a metric: JobGetAvailable, JobSetStateIfRunningMany, JobInsertFastMany, leader election, the rescuer and the cleaner. They show up only as pool acquires, OTel spans and CloudSQL totals. When the pool saturates, no metric tells River load apart from request load. #1664 states this caveat on PoolSaturated. S27 does not say what the accepted River database surface is.
Evidence
Research base: origin/main at e2c420456, 2026-10-25.
- River shares the pool:
internal/jobsriver/client.go:203(river.NewClient(riverdatabasesql.New(c.db), cfg)) andcmd/artifact-registry/wire_jobs.go:196-205. The constants are atinternal/jobsriver/client.go:36(defaultMaxWorkers = 25) and:40(defaultFetchPollInterval = 500 * time.Millisecond). - S27 justifies the shared pool with "One pool, one set of connection limits, one set of metrics" (
docs/specs/S27-background-jobs-foundation.md:186-187). It claims tracing is inherited through the pgxQueryTracer(:194-210). No metric attributes pool use to River. - LabKit has no seam for a second tracer.
postgres.NewWithConfigsetsconnCfg.Tracer = &queryTracer{...}(labkitv2@v2.63.2/postgres/client.go:207-209), andStartcopies that privateconnCfginto the pool (:271).NewOptionoffers onlyWithInfraConfig,WithTracerandWithConnectionPool(:324-349). This is the same at the latest tag,v2.65.1. pgxv5.11.0shipsmultitracer, so chaining is a small upstream change. - An application-side wrapper does not avoid the upstream change. A second
*sql.DBover a wrappedstdlibconnector adds its owndatabase/sqlconnection layer on top of the pgx pool. It also missesInsertTxon a caller's transaction (cmd/artifact-registry/bulk_enqueuer.go:187), because that runs on the application's*sql.Tx. That option was rejected.
Fix direction
- Amend S27
#### Database connectionanddocs/dev/observability.mdto record that pool stats, OTel spans andjobs_in_flight{backend="river"}(docs/dev/observability.md:71) are the accepted River database surface today. - Say what that surface cannot show: River's share of pool time and of database time. Point to #1722 as the follow-up that adds per-query counters once LabKit can chain a tracer.
- File the LabKit feature request below in the LabKit project (operator action; the text is ready), and link it from this issue and from #1722.
LabKit feature request (draft)
Title: postgres: let callers chain an additional
pgx.QueryTraceralongside the OTel tracer
postgres.NewWithConfigsetsconnCfg.Tracer = &queryTracer{...}(v2/postgres/client.go:207-209at v2.63.2, unchanged at v2.65.1), andStartcopies the privateconnCfginto the pool (:271).NewOptionoffersWithInfraConfig,WithTracerandWithConnectionPoolonly (:324-349), so a service cannot add its ownpgx.QueryTracerwithout losing LabKit's.Ask: a
postgres.WithQueryTracer(pgx.QueryTracer)option (repeatable, or variadic) that LabKit chains with its own tracer throughgithub.com/jackc/pgx/v5/multitracer. LabKit's tracer stays first and is always present.Use case: Artifact Registry runs River over the shared pool and wants to count River's own statements (which start with
-- name: <Query>) by query, without re-timing statements the application already times by name.Size: about 50-100 lines plus a test that asserts both tracers see
TraceQueryStart/TraceQueryEnd.
Acceptance criteria
- S27
#### Database connectionanddocs/dev/observability.mdrecord that pool stats, OTel spans andjobs_in_flight{backend="river"}are the River database surface, and name what they cannot show. - The docs point to #1722 for River query counters. They describe the current state only, with no promise of a date.
- The LabKit feature request is filed and linked from this issue and #1722 (operator action, outside the smurfit unit). Covered by gitlab-org/labkit#154.
- S27 stays
Status: Approved.
Size and fit
About 30-60 lines of docs. This fits one smurfit unit.
Dependencies
None. #1664 does not need this issue. It documents the "may be job load" caveat on PoolSaturated.