docs(specs): add S27 background jobs foundation
Summary
Adds the S27 spec — the background-jobs foundation that operationalizes the hybrid (River + asynq) architecture finalized in ADR-006 §1, and the operational dev doc engineers consult when adding job types.
What this spec covers
- Two backend-specific clients, not a unified abstraction.
jobsriver.Client(PostgreSQL-backed; transactional enqueue) andjobsasynq.Client(Redis-backed; high-throughput, fire-and-forget) share a smalljobspackage for cross-cutting types only (Argsinterface,Queue,Backend, shared error sentinels,1 MiBpayload cap). Backend selection is at the call site — which client'sEnqueue/Register*you invoke. No marker interfaces, no silently-dropped options, no abstraction leaks. (Resolved during review — see## Resolutionsin the spec.) - Backend selection rule is owned by
docs/dev/background-jobs.md(loss-consequence framing, River-vs-asynq criteria, default-to-asynq, lifecycle examples). S27 keeps the formal contract; the dev doc keeps the operational guidance. - River runs in poll-only mode (PgBouncer transaction pooling makes
LISTEN/NOTIFYunsafe). Shares the application's pgxpool via LabKit'spostgres.Client.Pool(). - River schema migrations are extracted into the existing goose pipeline as ordinary
*.sqlfiles; incremental extractor; CI freshness check;rivermigrate.Migrator.Migrateis not used at runtime. jobsriver.RegisterPeriodic[T]with auto-deduplication viaUniqueOpts{ByArgs, ByPeriod};IntervalandCronmutually exclusive (panic at registration); 5-field standard cron + descriptors viarobfig/cron/v3.ParseStandard. Periodic dispatch is leader-elected by River.- asynq trace context propagation via native
NewTaskWithHeaders. - Three queue tiers (
critical/default/low), fixed in code, operator-tunable per environment. - Unified queue-stats polling for both backends. One
*redislock.Client(built fromredis.jobs) shared by both clients; each poller holds its own lease on its own hash-tag-prefixed key (<prefix>:{metrics}:river_poller_lock,<prefix>:{metrics}:asynq_poller_lock). Lease + refresh, register-on-acquire / unregister-on-loss.metrics.enabledandmetrics.lease_durationconfigurable per backend. - Lifecycle integrates with S01 — both clients are LabKit
app.Components registered before the HTTP server (LIFO drains HTTP first, then jobs). Loader validates:app.Config.ShutdownTimeout >= server.shutdown.delay + server.shutdown.timeout + jobs.river.shutdown_timeout + jobs.asynq.shutdown_timeout. - Handler dependencies pattern. Plain Go — each feature package defines its own
Depsstruct; River workers hold dependency fields; asynq handlers capture state via a privatehandlersstruct; composition root assembles per-package deps. No DI framework. - Observability surface.
jobs_enqueued_total,jobs_processed_total,jobs_duration_seconds,jobs_in_flight,jobs_queue_depth,jobs_queue_oldest_age_seconds,jobs_errors_total(withunknowncatch-all bucket for forward compatibility),jobs_redis_health_check_failures_total. Metric prefix is owned by S03. - Out of Scope. Specific job types and schedules (owned by feature specs); migration of in-flight jobs across breaking-args changes (workloads own their drain plans); per-kind backlog metrics; runtime concurrency tuning; operational UIs.
Dev doc additions (docs/dev/background-jobs.md)
- Operating manual for engineers adding job kinds: routing, registration, transactional bind, enqueue options, periodic schedules, error handling, observability, pitfalls.
- Sidekiq capability mapping matrix — Sidekiq capability × River v0.35.1 × asynq v0.26.0 with verdict and pointers to relevant S27 sections, plus a Pro/Enterprise footer.
- Bumping River. Two-step happy path + the two CI-freshness-check failure modes with their fixes (with the non-obvious "don't rewrite the committed migration" rule called out).
- Spec authoring checklist — referenced from the
spec-authorskill.
Ownership boundaries
S27 defers Redis topology, TLS, pool tuning, and key-naming conventions (including the literal lock-key strings used by the metric pollers) to S05 (Distributed state). This MR consumes a single RedisConfig instance (redis.jobs) via two helpers: BuildAsynqRedisConnOpt (asynq's connection options) and the shared *redislock.Client used by both metric pollers. S27 mandates the Redis Cluster {metrics} hash-tag form on both lock keys; S05 chooses literal prefixes.
Until S05 is approved, the implementation MR ships a minimal provisional RedisConfig shape; S05's canonical definition supersedes it.
Metric registry / exposition is deferred to S03 (Observability).
Open question (transparent — needs resolution before implementation MR opens)
The spec asserts river.Client[pgx.Tx] (River's pgx-native driver), but the application query layer uses jet over database/sql. A *sql.Tx and a pgx.Tx from the same pool are different transactions on different connections — they cannot share atomicity. Three resolution paths are documented in ## Open Questions in the spec; resolution lands before any implementation MR.
Other deps shipped
bsm/redislock(Apache-2.0) — added todocs/dev/go-libraries.md. May be superseded by S05 if S05 standardizes a shared distributed-locking library.- Vale rule
local.SectionSign(under.vale/local/, level=error) — bans§repo-wide;lint:markdownblocks any reintroduction.
Out-of-scope from this MR: no specific job types are registered here — feature specs (S18, S20, S21, S22) introduce their own kinds on top of this foundation.