Retry observability contract

Context

Phase C of the canopy-test-lib port (!318 (merged), refs #436 (closed)) landed the fault-injection harness + 4 chaos tests under `crates/canopy-test-lib/tests/evil_proxy_test.rs`. The chaos tests use `SpanCapture::assert_span_emitted(target)` to verify production code emits the operator-runbook tracing signals operators rely on.

`circuit_breaker` instrumentation landed in-MR (`crates/canopy-api/src/circuit_breaker.rs::record_failure` now emits a `target: "circuit_breaker"` warn event on the closed→open transition). The other three span targets the chaos tests assert against are pre-existing gaps:

  1. `retry` — `inbox_dedup_at_100_percent_failure` chaos test expects a `target: "retry"` event when typed-client calls retry after 503s. canopy-test-lib's `TestClient` is one-shot today — retry behaviour lives at the orchestrator level (`services/canopy-eligibility/src/orchestrator.rs` + `canopy-api` retry middleware). Add a `tracing::info!(target: "retry", attempt, backoff_ms, ...)` event at the retry-attempt site.

  2. `jwks` — `jwks_rotation` chaos test expects a `target: "jwks"` event when the JWKS cache falls back to a cached key on Keycloak 5xx. `crates/canopy-auth/src/jwks.rs` already emits via `info!/warn!` but with the default target (`canopy_auth::jwks`) — change to explicit `target: "jwks"` on the refresh + fallback sites.

  3. `outbox` — `outbox_catches_up` chaos test expects a `target: "outbox"` event when the outbox-drainer retries after RabbitMQ outage. `crates/canopy-mq/src/outbox_drainer.rs` already emits `info!/warn!` extensively but without explicit `target: "outbox"` — adjust the retry-iteration spans.

Acceptance

  • `canopy_api::retry` (or wherever retry logic lives) emits a `target: "retry"` event on each retry attempt with `attempt` + `backoff_ms` fields.
  • `canopy_auth::jwks::refresh_*` emit `target: "jwks"` events on cache hit, miss, and forced-refresh paths.
  • `canopy_mq::outbox_drainer::drain_iteration` emits a `target: "outbox"` event on each retry.
  • All 4 chaos tests pass under `cargo nextest run -p canopy-test-lib --run-ignored only` against a live devstack.

Why not Phase C in-MR

Each instrumentation site needs the source change + a verification path. Doing one (circuit_breaker) was 5 minutes; doing all four with verification approached 1-2 hours of cross-crate audit, which the in-MR scope no longer supports. The chaos tests ship as contracts documenting the observability surface; they go green when this issue's checklist completes.

Refs

  • !318 (merged) (Phase C in-flight)
  • `crates/canopy-test-lib/tests/evil_proxy_test.rs` (the 4 chaos tests)
  • `crates/canopy-test-lib/src/observability.rs::SpanCapture::assert_span_emitted`
  • `crates/canopy-api/src/circuit_breaker.rs::record_failure` (the in-MR example)