Improve Redaction Service / Auth Scalability in Monolith Components

Problem

The Knowledge Graph's RedactionService fires hundreds of SQL queries per batch authorization request due to per-resource authorization lookups (N+1 on project_authorizations, members, saml_providers, License.current). Additionally, GKG query execution blocks Puma worker threads for 500ms-8s per request while the bidirectional gRPC stream runs.

Solution

Five MRs across two repos addressing query performance and thread utilization.

MR What it does
Add Additional Preloads to RedactionService (gitlab-org/gitlab!229374 - merged) Eager-load associations in PRELOAD_ASSOCIATIONS (72% query reduction)
Add batch authorization cache pre-seeding to Re... (gitlab-org/gitlab!229378 - merged) Batch pre-seed authorization caches + License.current SafeRequestStore
Use workhorse pattern for Orbit graph queries (gitlab-org/gitlab!229394 - merged) Move GKG queries from Puma to Workhorse via SendData/Injecter pattern
feat(proto): publish Go protobuf stubs for work... (!732 - merged) Go protobuf stubs for Workhorse integration
docs(adr): add workhorse query pattern (ADR 008) (!730 - merged) ADR 008: Workhorse query acceleration design doc

Related issues:

!229374: Eager-load associations

Added missing associations to PRELOAD_ASSOCIATIONS so they are batch-loaded upfront in the existing includes() call. DeclarativePolicy conditions access :organization, :saml_provider, :namespace, :assignees, :author, and :system_note_metadata during Ability.allowed? checks, but these were not being eagerly loaded, causing N+1 lazy-load queries per resource.

Results (215 resources, cold caches):

Metric Before After Reduction
Mixed batch queries 430 118 72%
system_note_metadata 50 (N+1) 1 (batch) 98%
licenses 217 55 75%

!229378: Batch authorization cache pre-seeding

Before policy evaluation, RedactionService now collects all projects and groups from loaded resources in a single pass, then runs ProjectPolicyPreloader and GroupPolicyPreloader to batch-load max_access_level into RequestStore. This replaces per-resource N+1 queries on project_authorizations and members tables.

Also wraps License.current in SafeRequestStore to avoid repeated JSON deserialization and AR object allocation on every call (each call parses JSON and allocates two License AR objects, ~800 unnecessary allocations per 50-project batch).

Postgres.ai results (session webui-150381, production clone):

Query Batch (1 query) N+1 per call N+1 x50 total Improvement
project_authorizations 23.9ms 1.6ms 95ms 2.5x, 49 fewer round trips
saml_providers 0.07ms 0.9ms 26ms 371x, 28 fewer round trips
group member access 85.7ms 167ms 3,340ms 39x, 19 fewer round trips
License.current 0ms (cached) 1.4ms 122ms 86 JSON parses eliminated

!229394: Workhorse query acceleration

Moves the bidirectional gRPC stream between Rails and GKG from Puma worker threads to Workhorse goroutines using the SendData/Injecter pattern (same pattern used by Gitaly for blob serving).

How it works:

  1. Rails receives the query request, authenticates the user, builds a JWT
  2. Rails responds with a Gitlab-Workhorse-Send-Data header containing the GKG connection details
  3. Workhorse intercepts the header, opens a bidirectional gRPC stream to GKG
  4. During the stream, GKG sends RedactionRequired messages back to Workhorse
  5. Workhorse calls a Rails internal API (POST /api/v4/internal/orbit/redaction) to check permissions
  6. Workhorse sends authorized resource IDs back to GKG, which filters results
  7. Workhorse writes the final JSON response to the client

Impact: Puma worker blocking drops from 500ms-8s (full query duration) to two short calls (~10-50ms each: initial auth + redaction callback). The query execution and streaming happen entirely in Workhorse goroutines.

Implementation:

  • workhorse/internal/orbit/: 3 Go files (client.go, sendquery.go, redaction.go) + generated proto stubs
  • ee/lib/gitlab/workhorse.rb: send_orbit_query method
  • ee/lib/api/internal/orbit.rb: POST /api/v4/internal/orbit/redaction endpoint
  • ee/app/controllers/dashboard/orbit/data.rb: ORBIT_WORKHORSE_ENABLED env var gate
  • All 5 query types pass E2E: search, traversal, aggregation, path_finding, neighbors
  • MCP query_graph tool also routed through Workhorse via McpId param

Supporting MRs:

Edited by Michael Angelo Rivera