feat(server): query result cache for cursor pagination

What does this MR do and why?

Part of #248 (closed), Depends on !699 (merged) and !698 (merged). Every page of a cursor query currently runs the full server pipeline: ClickHouse query, Rails authorization exchange, redaction, hydration. For an agent paging through 85 results in pages of 20, that's 5 identical round trips to ClickHouse and 5 bidirectional gRPC redaction exchanges with Rails — all producing the same authorized result set, just sliced differently.

This MR adds an in-memory query result cache at the run_query level. The first page runs the full pipeline and caches the complete authorized result (post-redaction, post-hydration, pre-cursor-slicing). Subsequent pages hit the cache and just apply the cursor window — no ClickHouse, no Rails, no redaction.

Page 1:  full pipeline → cache → slice [0..20] → return
Page 2:  cache hit     →         slice [20..40] → return
Page 3:  cache hit     →         slice [40..60] → return

The cache uses moka for lock-free concurrent access with automatic TTL eviction. Cache key is (user_id, hash(query_json without cursor)) so different pages of the same query share an entry.

TTL: 60 seconds — short enough that authorization changes propagate, long enough for multi-page browsing.

Capacity: 16,384 entries with LRU eviction. Cursor slicing moved from OutputStage to run_query so the full result is available for caching before the cursor window is applied.


Memory utilization analysis of LRU cache

Each entry is one full PipelineOutput — up to 1000 rows with hydrated properties. A typical search returning 30 users with 5-10 string columns runs ~2-5 KB.

Entries 1 KB/entry 2 KB/entry 5 KB/entry 10 KB/entry
256 256 KB 512 KB 1.25 MB 2.5 MB
512 512 KB 1 MB 2.5 MB 5 MB
1,024 1 MB 2 MB 5 MB 10 MB
2,048 2 MB 4 MB 10 MB 20 MB
4,096 4 MB 8 MB 20 MB 40 MB
8,192 8 MB 16 MB 40 MB 80 MB
16,384 16 MB 32 MB 80 MB 160 MB

At the configured cap with typical entries, we're looking at 30-80 MB. The 160 MB upper bound assumes every cached query returns the maximum 1000 rows with large hydrated payloads — unlikely in practice since most queries use the default limit of 30.

Edited by Michael Usachenko

Merge request reports

Loading
Loading