## Problem Any orbit query that returns rows from a Project node alongside a source-code node (Branch, File, Definition, Directory, ImportedSymbol) fails with HTTP 502 from `/api/v4/orbit/query`. The session looks like a hang from the user's perspective: the agent retries the same call, sees `Tool execution was interrupted`, and eventually gives up. The 502 is injected by Workhorse after the redaction callback returns 400 from Rails. Rails rejects the redaction request because Workhorse sends two entries with the same `resource_type: project` (one with ability `read_project`, one with `read_code`), and the endpoint at [orbit.rb#L81](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/internal/orbit.rb#L81) treats that as malformed input. ## Why duplicates get sent (this is not a GKG bug) The duplicate `resource_type` is the ontology working as designed. Two sets of entity types deliberately share `resource_type: project` because they authorize against the parent project record but with different abilities: | Entity | resource_type | ability | |---|---|---| | `Project` | project | `read_project` | | `Branch`, `File`, `Definition`, `Directory`, `ImportedSymbol` | project | `read_code` | Those are two distinct authorization decisions. A Reporter on a private repository can have `read_project` but not `read_code`, in which case a Project node should be visible while its Branches stay redacted. GKG's `resource_checks()` correctly groups by `(resource_type, ability)` and emits two `ResourceCheck` entries for a query that touches both kinds of nodes. The bug lives in the wire format between GKG, Workhorse, and Rails, not in the ontology. ## Steps to reproduce 1. Run any query that selects a Project plus any source-code entity. The minimal payload from a real failed session (workflow 123): ```json { "query": { "limit": 20, "nodes": [ { "id": "p", "entity": "Project", "columns": ["id"], "node_ids": [1000000] }, { "id": "b", "entity": "Branch", "columns": ["id", "name", "is_default"] } ], "query_type": "traversal", "relationships": [ { "to": "p", "from": "b", "type": "IN_PROJECT", "direction": "outgoing" } ] } } ``` 2. Post it to the public REST surface: ```bash curl -sk -X POST "https://gdk.test:3443/api/v4/orbit/query" \ -H "PRIVATE-TOKEN: $PAT" -H "Content-Type: application/json" \ -d @query.json ``` 3. Observe HTTP 502 with body "Bad Gateway". 4. Tail `log/api_json.log` for the redaction callback: ``` POST /api/v4/internal/orbit/redaction status=400 resources=[ { resource_type: project, resource_ids: [1000000], ability: read_project }, { resource_type: project, resource_ids: [1000000], ability: read_code } ] api_error: ["{\"message\":\"400 Bad request - duplicate resource_type 'project'\"}"] ``` The same chain triggers from the agent path (orbit_agent/v1 workflow). The user-visible symptom is a session that hangs around 60 to 120 seconds and then surfaces `Tool execution was interrupted` for every subsequent call. ## Long-term fix: discriminator on the response side The current `ResourceAuthorization` proto carries only `resource_type`, and GKG's `is_authorized` matches on that one field with a first-hit `find()`. See [query_result.rs#L380-395](https://gitlab.com/gitlab-org/orbit/knowledge-graph/-/blob/main/crates/query-engine/types/src/query_result.rs#L380-395). Allowing duplicate `resource_type` entries in the response without a discriminator is unsafe. A row authorized under `read_project` could be matched against the entry computed for `read_code` (or vice versa), depending on iterator order. That is a privilege-escalation risk; do not ship a Rails-only "tolerate duplicates" hotfix. The fix is a coordinated three-layer change with `(resource_type, ability)` as the unit on the response side. GKG: add an `ability` field to `ResourceAuthorization`. Change `is_authorized` to match on `(resource_type, ability)` against the requesting node's auth config. Drop the first-hit fallback. ```protobuf message ResourceAuthorization { string resource_type = 1; string ability = 2; // NEW map<int64, bool> authorized = 3; } ``` Workhorse: pass the new `ability` field through from the JSON callback into the proto response. Stop relying on `resource_type` alone as the response key. Rails: accept duplicate `resource_type` entries when the `(type, ability)` pair is unique. Aggregate by `(type, ability)`, run `Authz::RedactionService` per pair, and return one entry per pair with the matching `ability` field set. The proto's existing `repeated string abilities` (request side) becomes vestigial under this design and can be retired in a separate cleanup. ## Temporary workaround Change `Project.redaction.ability` from `read_project` to `read_code` in [config/ontology/nodes/core/project.yaml](https://gitlab.com/gitlab-org/orbit/knowledge-graph/-/blob/main/config/ontology/nodes/core/project.yaml). All entities under `resource_type: project` then share the same ability and the duplicate disappears at the source. This is fail-closed: it tightens visibility on Project nodes (Reporters with only `read_project` lose access). Acceptable temporarily; track the loss as the reason to ship the long-term fix. ## Observability gaps that made this hard to debug This bug took longer to find than it should have because every layer obscured the cause. The Workhorse top-level error message says "Bad Gateway" even though Rails returned 200 to every visible request. The 400 was on the internal redaction callback, which only shows up in `api_json.log` at `severity: INFO` with the failing payload buried in the `params` field. Easy to miss when scanning for ERROR-level rows. The MCP go-sdk wraps the failure as `rejected by transport: ... Bad Gateway`, which reads like a network-level transport failure (connection refused, DNS, and so on). The actual cause was an in-process JSON-RPC 502 returned by Workhorse's badgateway round-tripper after the redaction callback errored. DWS logs say `RPC cancelled by client`, which sounds like the browser disconnected. The "client" here is Workhorse cancelling the gRPC stream on its way out; the browser was still attached. There is no correlation between the failing tool-call `request_id` and the redaction callback's correlation_id in logs. They share a top-level correlation ID, but Workhorse does not log the redaction request_id alongside the `Calling an MCP tool` lines. There is no metric for redaction failures. A dashboard for `/api/v4/internal/orbit/redaction` 4xx rate would have surfaced this on the first failing query. The agent-facing error is `Tool execution was interrupted. Please try running the tool again.` No diagnostic detail. The agent retries blindly until it gives up and the user sees `INPUT_REQUIRED`. Suggested follow-ups (separate from the fix): - Log the redaction request payload and response status alongside the corresponding MCP tool-call `request_id` in Workhorse. - Surface redaction 4xx counts as a Prometheus metric under the `knowledge_graph` feature category. - When Workhorse aborts a query because of a redaction error, propagate a structured error (not "Bad Gateway") through to DWS so the agent sees the actual reason. ## Acceptance criteria - [ ] Workaround applied: `Project.redaction.ability` set to `read_code`. The reproduction query above returns 200. - [ ] Long-term fix landed across GKG, Workhorse, and Rails. - [ ] After the long-term fix, revert the workaround so Project requires only `read_project` again. - [ ] At least one of the observability follow-ups landed (logging or metric). <!-- AI-Sessions dir: ~/.claude/projects/-home-arivera-gitlab-com/ b06c5247-bbbe-4218-9bc5-ab51939e9854.jsonl (2026-05-04) -->