Prometheus metrics for idempotency cache hit/miss/replay (op-infra Step 12 follow-up)

Context

IdempotencyCache (crates/canopy-api/src/idempotency.rs) has no Prometheus instrumentation. After op-infra Step 12 (persistent backend), there is real operational value in knowing:

  • Cache hit rate (canopy_idempotency_cache_hits_total{backend="memory|postgres"})
  • Cache miss / new-entry rate (canopy_idempotency_cache_misses_total{backend="..."})
  • Replay rate (canopy_idempotency_cache_replays_total{backend="..."}) — this is the same as hits for the live cache, but separating the counters makes per-route dashboards clearer.
  • TTL cleanup rows deleted per tick (canopy_idempotency_cache_ttl_deletes_total{backend="postgres"})
  • Postgres backend errors during the persistence path (canopy_idempotency_cache_persist_errors_total).

Acceptance criteria

  1. metrics::counter! increments at the four points above (use canopy_common::telemetry::metrics_registry()).
  2. Add a service label populated from canopy-common or the existing telemetry context so a single Prometheus instance can scrape every service and break down per-service hit rates.
  3. Document the metrics in docs/modules/ROOT/pages/observability.adoc (or wherever the metrics inventory lives — see existing canopy_db_pool_* gauges from Step 8 for the reference pattern).

Out of scope

  • Histogram of replay latency (separate dashboarding concern).
  • Per-route cardinality (would explode label space for the rate limiter — leave path-level analysis to access logs).

Source

Op-infra plan Step 12, code review of the implementation MR. Step 8 added pool metrics; Step 12 deserves the same treatment for the cache it ships.