feat: cross-process chaos observability harness

Description

Three of the four chaos tests in crates/canopy-test-lib/tests/evil_proxy_test.rs assert that production code emits tracing::Event targets fired inside devstack containers (canopy-auth JWKS refresh, canopy-mq outbox drainer running inside services). SpanCapture::install_scoped at crates/canopy-test-lib/src/observability.rs:120-130 uses tracing::subscriber::set_default, which is thread-local in the test process — it cannot observe events emitted in different processes. The current jwks_rotation and outbox_catches_up tests therefore pass for the wrong reasons (or fail flakily) and provide no real chaos-coverage signal.

This child decides and builds the test strategy for cross-process chaos observability. Candidates to evaluate:

  1. In-process production fixtures — spin up JwksProvider / OutboxDrainer directly in the test process behind an EvilLayer, the same way the retry chaos test (inbox_dedup_at_100_percent_failure) does. Pros: simple, SpanCapture works as-is. Cons: bypasses real devstack wiring.
  2. OTEL export — production code emits OTLP spans; tests collect them via an in-test OTEL collector. Pros: production-shaped, observability already partly wired (canopy-otel crate). Cons: more infrastructure, slower tests, exporter cost in prod.
  3. Log-capture via container stdout scraping — tests scrape JSON logs from running containers via Docker API. Pros: zero production code change. Cons: brittle, race-prone.

Output is a decision (likely option 1) plus the reusable harness modules. Children B and C consume the harness.

Acceptance Criteria

  • Decision document under docs/modules/ROOT/pages/adrs/adr-NNN-cross-process-chaos-observability.adoc with comparison + chosen approach + rationale.
  • Harness module (or modules) under crates/canopy-test-lib/src/ that supports asserting on tracing::Event emissions from production code under EvilProxy fault injection, with the chosen approach implemented end-to-end.
  • Reference example in the harness's rustdoc showing the canonical use shape.
  • Existing in-process chaos test (inbox_dedup_at_100_percent_failure at evil_proxy_test.rs:39-82) refactored to use the new harness if appropriate (so children B and C inherit a consistent pattern).
  • cargo xtask validate clean.

Context & References

  • Epic: &50 (closed) — Chaos observability contracts: cross-process capture + retry/JWKS/outbox
  • Sibling: #462 (closed) — Retry observability contract (in-process; closes via the retry middleware MR)
  • Sibling: Child B (JWKS contract) — consumes this harness
  • Sibling: Child C (Outbox contract) — consumes this harness
  • Affected files:
    • crates/canopy-test-lib/src/observability.rs:120-130 — current thread-local SpanCapture
    • crates/canopy-test-lib/tests/evil_proxy_test.rs:140-176, 184-220 — current blind chaos tests
    • crates/canopy-test-lib/src/ — harness lives here

Labels

priority::medium, service::shared-crates, program::infrastructure, type::feature, workflow::needs-spec