feat(compiler): replace keyset cursor with agent-driven pagination

What does this MR do and why?

NOTE: The keyset CNF predicate ((tp > x) OR (tp = x AND id > y)) is removed in this MR — but it's a valid performance optimization for single-table scans on (traversal_path, id) PKs and could be useful for QueryType::Search (for deep pages) or Hydration in the future.

Part of #248 (closed). Based off the research from ($5972795).

SQL-level pagination doesn't work for a graph query engine. Keyset pagination cannot track position in edge-space for traversals, aggregate groups, paths, or neighbor results.

On top of that, redaction happens after ClickHouse returns rows, so LIMIT/OFFSET operate in row-space while the client sees entity-space. A page of 20 rows might become 15 after redaction. Range/offset pagination had the same fundamental problem.

So in this MR, we replace both keyset and range pagination with an agent-driven cursor model that works for all query types and handles redaction correctly.

The agent sends cursor: {offset, page_size} alongside a limit that controls the ClickHouse fetch window. The server runs the full pipeline (query → authorize → redact → hydrate), then slices the authorized result set to [offset..offset+page_size]. Each request is stateless — the server re-runs the query and re-authorizes.

Data Flow

Agent:   { limit: 100, cursor: { offset: 0, page_size: 20 } }
Server:  CH(LIMIT 100) → auth → redact → 85 authorized → slice [0..20]
Return:  20 rows, has_more: true, total_rows: 85

Testing

To the data correctness harness, we added Requirement::Cursor — tests with cursor queries must call assert_node_count.

Test coverage (19 new tests):

  • Compiler validation: valid cursors, boundary cases, 6 negative cases (missing fields, page_size=0, overflow)
  • Data correctness: first/second/last/empty pages, cursor+filters, cursor+redaction, cursor on traversal, full page coverage (no overlap, no gaps)
  • Redaction E2E: cursor slices post-redaction set correctly

Follow-up MR(s)

  • Add cursor, has_more, total_rows to QueryMetadata proto
  • Server-side query caching with TTL to avoid re-running on page 2+
Edited by Michael Usachenko

Merge request reports

Loading
Loading