refactor: service-identity auth for service-to-service calls (replace JWT pass-through)

Context

Canopy currently uses JWT pass-through for service-to-service calls: a worker logs in via OIDC, gets a worker JWT, and that JWT is forwarded verbatim through every service hop (canopy-web → canopy-applications → canopy-eligibility → canopy-snap → canopy-rules). Every service authenticates the same JWT.

This is a known anti-pattern. The trust boundary doesn't match the deployment boundary: canopy-rules can't distinguish between canopy-snap calling it (legitimate) and a worker hitting `/v1/rules/evaluate` directly with their JWT (not legitimate but indistinguishable from the legitimate path).

External reviewer 2026-05-07 flagged the canopy-rules-specific instance: "any authenticated user can evaluate any ruleset and read past evaluations." Issue E0.5 / forthcoming MR ships per-endpoint role gating as the tactical fix. This issue captures the structural fix: replace pass-through with service-class JWTs end-to-end.

Pattern

Same architectural pattern as OpenStack service accounts (Keystone), AWS IAM roles + STS AssumeRole, GCP service accounts, Kubernetes service accounts + projected tokens, and SPIFFE/SPIRE. IETF specifies it as OAuth 2.0 `client_credentials` grant (RFC 6749 §4.4) plus optional Token Exchange (RFC 8693) for delegation.

Shape

Each canopy service gets:

  • A service-class identity in the OIDC IdP (e.g. Keycloak service account with `aud: canopy-service`)
  • A `client_id` + `client_secret` provisioned via SOPS-encrypted secrets (ADR-017)
  • Bootstrap-time mint of its service JWT via `client_credentials` grant against the discovered token endpoint (ADR-018-style discovery is already wired via #422 (closed))
  • In-memory cache + refresh-before-expiry logic, same shape as #411 (closed)'s BFF token refresh
  • canopy-auth distinguishes JWT types by audience claim (`aud: canopy` = worker, `aud: canopy-service` = service)
  • On-behalf-of plumbing (RFC 8693-style, or simpler `X-Canopy-OBO` header) for audit attribution

Worker UX is unchanged. The change is entirely at the trust boundaries between services.

Why filed as a separate refactor (not folded into E0.5)

Comparable in scope to the OIDC pluggability refactor (#422 (closed) / MR !215 (merged)) or the BFF token refresh (#411 (closed) / MR !216 (merged)). Touches:

  • canopy-auth (new audience-based policy)
  • Every service that calls another service (snap, tanf, medicaid, caps, wic, eligibility, web, applications)
  • canopy-cli (admin-class identity)
  • Devstack Keycloak provisioning (`xtask/src/devstack_guard.rs`)
  • Audit attribution logic
  • Test fixtures and integration tests across all of the above

Multi-MR effort, warrants its own ADR (proposed: ADR-019 "Service identity for service-to-service auth").

Acceptance criteria

  • ADR-019 drafted + accepted
  • Per-service Keycloak service accounts provisioned in devstack
  • Each service mints + caches its service JWT at startup
  • canopy-auth audience-aware policy
  • On-behalf-of plumbing across all service-to-service hops
  • Audit attribution updated to track `actor_id` (service) + `subject_id` (worker via OBO)
  • Worker JWT direct hits to internal services rejected (e.g., direct call to canopy-rules with a worker JWT returns 401)
  • All existing tests pass; new fixtures for service-JWT-bearing tests
  • CHANGELOG entry citing pattern + analogy to OpenStack/AWS/SPIFFE conventions
  • CLAUDE.md identity section updated

Dependencies

  • #422 (closed) (OIDC pluggability) — landed; provides discovery infrastructure
  • ADR-017 secrets at rest — landed; provides credential storage path
  • No other open work blocks this

Related

  • External review 2026-05-07 — flagged the canopy-rules-specific gap
  • Plan filed during E0.5 design discussion 2026-05-07