docs(specs): add S27 background jobs foundation

Summary

Adds the S27 spec — the background-jobs foundation that operationalizes the hybrid (River + asynq) architecture finalized in ADR-006 §1, and the operational dev doc engineers consult when adding job types.

What this spec covers

  • Two backend-specific clients, not a unified abstraction. jobsriver.Client (PostgreSQL-backed; transactional enqueue) and jobsasynq.Client (Redis-backed; high-throughput, fire-and-forget) share a small jobs package for cross-cutting types only (Args interface, Queue, Backend, shared error sentinels, 1 MiB payload cap). Backend selection is at the call site — which client's Enqueue / Register* you invoke. No marker interfaces, no silently-dropped options, no abstraction leaks. (Resolved during review — see ## Resolutions in the spec.)
  • Backend selection rule is owned by docs/dev/background-jobs.md (loss-consequence framing, River-vs-asynq criteria, default-to-asynq, lifecycle examples). S27 keeps the formal contract; the dev doc keeps the operational guidance.
  • River runs in poll-only mode (PgBouncer transaction pooling makes LISTEN/NOTIFY unsafe). Shares the application's pgxpool via LabKit's postgres.Client.Pool().
  • River schema migrations are extracted into the existing goose pipeline as ordinary *.sql files; incremental extractor; CI freshness check; rivermigrate.Migrator.Migrate is not used at runtime.
  • jobsriver.RegisterPeriodic[T] with auto-deduplication via UniqueOpts{ByArgs, ByPeriod}; Interval and Cron mutually exclusive (panic at registration); 5-field standard cron + descriptors via robfig/cron/v3.ParseStandard. Periodic dispatch is leader-elected by River.
  • asynq trace context propagation via native NewTaskWithHeaders.
  • Three queue tiers (critical/default/low), fixed in code, operator-tunable per environment.
  • Unified queue-stats polling for both backends. One *redislock.Client (built from redis.jobs) shared by both clients; each poller holds its own lease on its own hash-tag-prefixed key (<prefix>:{metrics}:river_poller_lock, <prefix>:{metrics}:asynq_poller_lock). Lease + refresh, register-on-acquire / unregister-on-loss. metrics.enabled and metrics.lease_duration configurable per backend.
  • Lifecycle integrates with S01 — both clients are LabKit app.Components registered before the HTTP server (LIFO drains HTTP first, then jobs). Loader validates: app.Config.ShutdownTimeout >= server.shutdown.delay + server.shutdown.timeout + jobs.river.shutdown_timeout + jobs.asynq.shutdown_timeout.
  • Handler dependencies pattern. Plain Go — each feature package defines its own Deps struct; River workers hold dependency fields; asynq handlers capture state via a private handlers struct; composition root assembles per-package deps. No DI framework.
  • Observability surface. jobs_enqueued_total, jobs_processed_total, jobs_duration_seconds, jobs_in_flight, jobs_queue_depth, jobs_queue_oldest_age_seconds, jobs_errors_total (with unknown catch-all bucket for forward compatibility), jobs_redis_health_check_failures_total. Metric prefix is owned by S03.
  • Out of Scope. Specific job types and schedules (owned by feature specs); migration of in-flight jobs across breaking-args changes (workloads own their drain plans); per-kind backlog metrics; runtime concurrency tuning; operational UIs.

Dev doc additions (docs/dev/background-jobs.md)

  • Operating manual for engineers adding job kinds: routing, registration, transactional bind, enqueue options, periodic schedules, error handling, observability, pitfalls.
  • Sidekiq capability mapping matrix — Sidekiq capability × River v0.35.1 × asynq v0.26.0 with verdict and pointers to relevant S27 sections, plus a Pro/Enterprise footer.
  • Bumping River. Two-step happy path + the two CI-freshness-check failure modes with their fixes (with the non-obvious "don't rewrite the committed migration" rule called out).
  • Spec authoring checklist — referenced from the spec-author skill.

Ownership boundaries

S27 defers Redis topology, TLS, pool tuning, and key-naming conventions (including the literal lock-key strings used by the metric pollers) to S05 (Distributed state). This MR consumes a single RedisConfig instance (redis.jobs) via two helpers: BuildAsynqRedisConnOpt (asynq's connection options) and the shared *redislock.Client used by both metric pollers. S27 mandates the Redis Cluster {metrics} hash-tag form on both lock keys; S05 chooses literal prefixes.

Until S05 is approved, the implementation MR ships a minimal provisional RedisConfig shape; S05's canonical definition supersedes it.

Metric registry / exposition is deferred to S03 (Observability).

Open question (transparent — needs resolution before implementation MR opens)

The spec asserts river.Client[pgx.Tx] (River's pgx-native driver), but the application query layer uses jet over database/sql. A *sql.Tx and a pgx.Tx from the same pool are different transactions on different connections — they cannot share atomicity. Three resolution paths are documented in ## Open Questions in the spec; resolution lands before any implementation MR.

Other deps shipped

  • bsm/redislock (Apache-2.0) — added to docs/dev/go-libraries.md. May be superseded by S05 if S05 standardizes a shared distributed-locking library.
  • Vale rule local.SectionSign (under .vale/local/, level=error) — bans § repo-wide; lint:markdown blocks any reintroduction.

Out-of-scope from this MR: no specific job types are registered here — feature specs (S18, S20, S21, S22) introduce their own kinds on top of this foundation.

Edited by Suleimi Ahmed

Merge request reports

Loading
Loading