feat(server): count items for up to 100 paths in one call

What does this MR do and why?

The Orbit overview page sends one item count request for each group, so 50 groups cost 50 gRPC calls. With this change, one GetItemCounts call counts 1 to 100 paths. Each path still runs the same count query as today. Design: snippet 6070846.

How it works

Proto (additive). A request with traversal_paths gets one namespaces entry per path, in request order. Field 1 traversal_path is now [deprecated = true]. The server still accepts field 1, so Rails that sends it keeps working during the rollout. A request with both fields is invalid.

message GetItemCountsRequest {
  string traversal_path = 1 [deprecated = true];
  repeated string traversal_paths = 2;  // 1 to 100
}
message GetItemCountsResponse {
  repeated DomainItemCount domains = 1;
  repeated NamespaceItemCounts namespaces = 2;  // request order
}

Queries. Each distinct path gets one count query. For one path, the SQL is byte-identical to main, with uniqIf kept. Up to 8 queries run at once (buffer_unordered(MAX_CONCURRENT_COUNT_QUERIES)).

Settings. Each count query sends three ClickHouse settings:

Setting	Why
enable_parallel_replicas=1	A large path reads on all replicas, not one.
parallel_replicas_min_number_of_rows_per_replica=50000000	Small paths stay on one replica, so they get no extra latency.
cancel_http_readonly_queries_on_client_close=1	A query stops when the gRPC caller leaves.

Cache and errors. The count query keeps the configured query cache switch and sets query_cache_ttl to 5 s. The production default caches every query for 10 s, so counts drop to 5 s. A failed path query is logged, and that path is left out of namespaces. The call fails only when every path fails. GetGraphStatus keeps its zeros on error, so its output does not change.

The docs change moves to a separate docs MR, because of the repository rule for code MRs.

Related Issues

Related to #263

• Rails side: gitlab-org/gitlab!264743 (merged) batches itemCounts and depends on the release that contains this MR.
• Docs: !3054 (closed).

Testing

• Container tests (run-container-tests.sh -E 'test(/server::(status|query_cache)::/)'): 6/6, three runs in a row.
• The tests check one plain query per path with the parallel replica settings, and 100 paths as 100 queries.
• They also check nested paths, a role per path, request order with a duplicate, the 5 s TTL, the request field, and a missing table.
• cargo nextest run -p orbit-server: 250/250. Strict clippy (--all-features -D warnings) passes.
• vm-2 e2e: skill validation passes 12/12. For 50 paths, the gRPC counts are byte-identical to main. One GraphQL memberGroups(first: 50) request returns 50 groups, and all 50 equal REST graph_status.

Performance Analysis

A first draft counted all paths in one batched query. In production, that query hit the 30 s ClickHouse limit for requests with several large groups, so this MR replaced it.

Production numbers for one per-path count query, with the cache off, follow.

Path	Without the settings	With the settings	Replicas
Largest group (1.52 B rows)	6.1 to 6.9 s, 47 to 63 CPU-s	4.1 s, 67 CPU-s	3
Large group (0.51 B rows)	2.5 s, 21 CPU-s	1.5 s, 23 CPU-s	3
Median group (0.25 M rows)	0.08 to 0.33 s	0.27 s	1

For 100 paths with 5 large groups, the projected wall time is about 4.5 to 6 s, below the 10 s gRPC deadline. No single query takes more than about 4 s. The CPU is about 190 CPU-s, the same as 100 calls today, because the SQL is the same. Peak memory is about 170 MiB per query.

• This merge request does not introduce any performance regression. If a performance regression is expected, explain why.

Agent context: long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered

• Distinct paths are sorted. So the same set of paths gives the same SQL in any request order.
• Visibility runs once per role. With a 3,000-path token and 100 paths, it takes 0.70 ms per call; before, it took 12.8 ms.
• A single-node ClickHouse drops enable_parallel_replicas from the logged settings. The test therefore reads the row threshold, which is sent with it.
• The query log fixture waits until every cached query has a finish row before it reads query_cache_usage. ClickHouse writes that log after the client has the result, and one flush missed a row in 2 of 10 runs.
• On vm-2, 1 path takes 33 ms, 6 paths 47 ms, and 50 paths 313 ms, with the cache off.

%{all_commits}