### Problem to Solve We cannot answer basic questions about how code indexing went for a repository, a branch, or the fleet without reading logs and metrics dashboards. "Did this repo index, and if not, why?", "which files got dropped?", "how many branches are stuck?" have no queryable answers today. On .com an SRE can dig through Kibana. A self-managed or air-gapped admin cannot, and neither can we on their behalf. The two places this data lives today are both wrong for the job: - gl_file.reason is a column on a versioned graph table. Rows are tombstoned on every re-index and the whole table is dropped when its schema version leaves the retention window. There is no history, exactly when history is what a debugging admin needs. - Per-branch state is success-only (code_indexing_checkpoint). A branch that times out, dead-letters, or never starts leaves no trace at all. !2166 tried to fix this with a gl_repository graph node and was closed for the same reason gl_file.reason is wrong: graph tables are the customer query surface, and diagnostics need a lifecycle the graph does not have. ### Proposed Solution Store indexing outcomes in two unversioned auxiliary ClickHouse tables, and serve them to instance admins through a new gRPC RPC that the gitlab:orbit:info rake task (gitlab-org/gitlab!246076) can call. Unversioned tables survive schema migrations and GC (same mechanism as namespace_storage_snapshot), so diagnostic history outlives re-indexes and version bumps. Decisions already made: - gl_file.reason is removed. The internal table is the single home for per-file outcomes. The FileReason enum stays as the producer for both the table and the existing file_faults metrics. - Cleanup is TTL-only (90 days). The namespace-deletion sweep does not cover these tables; paths of deleted projects linger until the TTL expires. Documented, accepted. - No free-text columns. Every status and reason value comes from a Rust enum with a single as_str producer and a label-stability test, the same rule FileReason already follows. This is also the error-string hygiene story: nothing unbounded can leak through the endpoint. - Evolving unversioned table schemas is a known gap and gets its own design issue. These tables should be narrow enough to get right on the first pass. #### 1. Tables (config/ontology/schema.yaml, auxiliary_tables) ```yaml - name: code_indexing_branch_events versioned: false include_system_columns: false engine: MergeTree columns: - {name: project_id, type: int64, codec: ["zstd(1)"]} - {name: branch, type: string, codec: ["zstd(1)"]} - {name: traversal_path, type: string, codec: ["zstd(1)"]} - {name: task_id, type: int64, codec: ["zstd(1)"]} - {name: status, type: string} # indexing | indexed | failed - {name: fail_reason, type: string, default: "''"} # timeout | transient | permanent - {name: commit, type: string, default: "''"} - {name: started_at, type: timestamp, codec: ["delta(8)", "zstd(1)"]} - {name: duration_ms, type: int64, default: "0"} order_by: [project_id, branch, started_at] ttl: started_at + INTERVAL 90 DAY - name: code_indexing_file_events versioned: false include_system_columns: false engine: MergeTree columns: - {name: project_id, type: int64, codec: ["zstd(1)"]} - {name: branch, type: string, codec: ["zstd(1)"]} - {name: task_id, type: int64, codec: ["zstd(1)"]} - {name: path, type: string, codec: ["zstd(1)"]} - {name: reason, type: string} # FileReason::Display strings - {name: occurred_at, type: timestamp, codec: ["delta(8)", "zstd(1)"]} order_by: [project_id, branch, task_id, path] ttl: occurred_at + INTERVAL 90 DAY ``` Both are append-only MergeTree event logs. Branch events get one row per status transition (start, then terminal), so attempt history is a plain range scan. File events are written only for files with a non-empty reason; clean files are the vast majority and are not logged. #### 2. Indexer write path New store next to checkpoint.rs, parameterized inserts, best-effort (a failed diagnostics write logs a warning and never fails the task). This is the store already written for !2166, retargeted at an unversioned table and stripped of the graph-node parts: ```rust // crates/indexer/src/modules/code/diagnostics.rs pub enum BranchStatus { Indexing, Indexed, Failed } pub enum BranchFailReason { Timeout, Transient, Permanent } // single as_str() producer per enum + label-stability test, as FileReason does pub struct ClickHouseDiagnosticsStore { client: ArrowClickHouseClient } impl ClickHouseDiagnosticsStore { pub async fn record_branch_event(&self, ev: BranchEvent) -> Result<()>; pub async fn record_file_events( &self, task: TaskRef, reasons: &[(String, FileReason)], ) -> Result<()>; } ``` Call sites mirror !2166: an indexing row after the per-branch lock is acquired (handler.rs, next to the KV record_start) and a terminal row at completion. File events come from the per-path reason map the pipeline already builds (pipeline.rs merges stream skips, parse skips, and faults into one map before the File batch is written). Removing gl_file.reason touches: the ontology property on file.yaml, the FileRow column in linker/graph.rs, the local DDL in graph_local.sql, and a schema version bump with a ledger entry. #### 3. Endpoints (two new RPCs on KnowledgeGraphService) Request fields are plain filters, the same style GetGraphStatus uses. There is deliberately no query language on the wire: canned queries live server-side, and ad-hoc analysis happens client-side (see export below). If server-side ad-hoc querying is ever needed, the path is the existing orbit DSL gaining an admin-gated internal domain (separate design issue), never a second grammar. ```proto // Formatted summary for the rake task's info sections. rpc GetIndexingDiagnostics(GetIndexingDiagnosticsRequest) returns (GetIndexingDiagnosticsResponse); // File download: streams encoded bytes, no per-row proto mapping. rpc ExportIndexingDiagnostics(ExportIndexingDiagnosticsRequest) returns (stream ExportChunk); message GetIndexingDiagnosticsRequest { optional int64 project_id = 1; // omit for fleet-wide branch summary optional string branch = 2; bool include_file_events = 3; // only honored when project_id is set optional string since = 4; // RFC3339 uint32 limit = 5; } message BranchEvent { int64 project_id = 1; string branch = 2; string status = 3; string fail_reason = 4; int64 task_id = 5; string commit = 6; string started_at = 7; uint64 duration_ms = 8; } message FileEvent { string path = 1; string reason = 2; int64 task_id = 3; string occurred_at = 4; } message GetIndexingDiagnosticsResponse { repeated BranchEvent branch_events = 1; repeated FileEvent file_events = 2; } message ExportIndexingDiagnosticsRequest { optional int64 project_id = 1; optional string branch = 2; optional string since = 3; Format format = 4; // PARQUET | NDJSON | CSV Dataset dataset = 5; // BRANCH_EVENTS | FILE_EVENTS } message ExportChunk { bytes data = 1; } ``` Export is a byte stream for two reasons: fleet-wide file events can exceed the default 4MB gRPC message cap, and Parquet lets the handler reuse the Arrow results it already gets from ClickHouse (arrow-rs ships a Parquet writer; row groups flush incrementally, so memory stays constant). The Ruby side is a byte sink that appends chunks to a file. Handlers follow GraphStatusService: lower filters to the compiler's typed AST, compile via codegen, bind parameters, fetch_arrow. No hand-written SQL strings. Table names resolve through ontology.auxiliary_tables(); unversioned tables need no version-prefix resolution, so the handlers stay correct mid-migration. Auth is a hard admin gate on both RPCs: ```rust let ctx = extract_request_context(&request, &self.jwt_validator)?; if !ctx.claims.admin { return Err(Status::permission_denied("instance admin required")); } ``` Precedent: authorize_traversal_path already short-circuits on claims.admin, and GetClusterHealth already accepts a bare user-actor JWT. No new auth machinery. #### 4. Rails side Proto changes ship through the existing gitlab-gkg-proto gem pipeline (clients/protogem). The rake task grows two things, both reusing the ORBIT_INFO_USER actor from gitlab-org/gitlab!246076. Separate monolith MR. An "Indexing internals" section in InfoService: fleet branch-status summary by default, per-project file events in extended mode. An export task that writes the streamed bytes to a local file: ``` gitlab-rake "gitlab:orbit:export[/tmp/orbit-diagnostics.parquet]" ``` Parquet is the default format because the admin (or a support engineer handed the file) can then run arbitrary SQL locally without any server-side query surface: ``` duckdb -c "select reason, count(*) from 'orbit-diagnostics.parquet' group by 1 order by 2 desc" ``` This is the same DuckDB workflow as orbit local, and it is why the RPCs can stay filter-only. #### Relation to #1106 The repo-level .orbit/config.yml rules from #1106 settle excluded files with a reason during filtering. Those reasons land in code_indexing_file_events through the same write path, so a tenant can change their config and verify the effect with the same rake task. That closes the control-and-verify loop #1110 describes. #### Delivery order 1. Tables + branch event writes (indexer only, no schema bump). 2. File event writes + gl_file.reason removal (schema bump + ledger). 3. RPCs + gem release. 4. Rails InfoService section + export task (monolith MR). Each step ships on its own. <details> <summary><b>Agent context</b> — research notes and rejected alternatives</summary> #### Why graph tables were rejected as the home for diagnostics Versioning: every node/edge table is created per schema version with a v{N} prefix (crates/indexer/src/schema/version.rs:146), rebuilt by re-indexing after a migration, and dropped by GC when the version leaves the keep-set (crates/indexer/src/orchestrator/scheduled/migration_completion.rs:100). Within a version, re-indexing a branch tombstones prior File rows (stale_data_cleaner.rs:81). Checkpoints are the only state seeded across versions (migration.rs:71). So reason columns are current-state only and vanish on the two occasions an admin most wants history: after a failed re-index and after a migration. Query surface: named queries and the DSL are user-permission-scoped (traversal paths compiled into SQL plus Rails redaction) and can only reach ontology entities. Fleet-wide admin queries are structurally impossible there, and auxiliary tables are invisible to the DSL. A separate admin RPC over internal tables avoids bending either system. namespace_storage_snapshot (schema.yaml auxiliary_tables, versioned: false, ReplacingMergeTree, TTL 400 days) is the structural template: an unversioned table created once at boot through generate_unversioned_objects (crates/query-engine/compiler/src/passes/codegen/ddl/mod.rs:89), snapshotted in config/graph_persistent.sql, excluded from GC. #### Auth constraints found All gRPC tokens are user-actor JWTs with required source_type, user_id, username (crates/gkg-server/src/auth/claims.rs:19). There is no service-account path on the gRPC surface; the reverse direction (GKG to Rails) uses a different scheme (gkg-indexer:code subject in the Gitlab-Orbit-Api-Request header). The smallest correct increment is a claims.admin gate, which also matches how a rake task run by an instance admin authenticates. build_security_context is not called at all; this endpoint bypasses the compiler entirely. #### Sanitation precedent graph_status replaces raw error strings with a generic message before returning them (graph_status/mod.rs:340). This spec avoids the problem by construction: enum-bounded columns mean the endpoint can only ever return known label strings. #### Rejected: diagnostic entities in the ontology, queried via the orbit DSL Registering the two tables as hidden ontology entities and querying them through ExecuteQuery was seriously considered: it reuses the grammar, and the security pass plus redaction exchange would give traversal-path scoping and per-resource Ability checks for free. It lost on the threat model. Rake execution requires shell on the Rails node, which is already instance-superuser (console access, signing key, full database); scoped per-user auth adds nothing on that path, and the admin gate covers the network path. The ontology route also depends on two new compiler capabilities (unversioned node tables and a hidden/admin-gated domain, roughly half of a 1.5-2.5k line estimate, with the risk concentrated in auditing the migration/GC machinery for unversioned nodes). If non-admin scoped access over the API ever becomes a requirement, this is the design to revisit, and the write side specced here does not change. #### Rejected: server-side ad-hoc query surfaces Raw SQL passthrough (rake sends SQL, GKG executes read-only against the two tables) was considered. Enforcing it safely needs either a dedicated ClickHouse user with SELECT grants on only these tables (a new secret to provision in every install) or readonly settings plus EXPLAIN QUERY TREE table-set validation (parser-adjacent, easy to get wrong; table functions like remote() and url() escape text-based allowlists). A structured query proto (filters, group_by, aggregates) was also considered and dropped: it is a bespoke query language in proto form. Parquet export plus local DuckDB gives full SQL with zero server-side surface, which removes the need for either. If server-side ad-hoc querying is ever justified, the design is an admin-gated internal domain in the ontology so the existing orbit DSL covers it; that gets its own design issue. #### Rejected: keep gl_file.reason alongside the file events table Keeping both means two homes for the same fact and an ontology property whose values can drift from the diagnostics table between re-indexes. The column also predates any consumer: no named query references it and it is not in File default_columns. Removing it costs one schema bump now and avoids a permanent consistency liability. #### Volume estimate Branch events: two rows per indexing task. File events: bounded by skip/fault counts, which the file_faults and files.skipped metrics show are a small fraction of files parsed. Both tables are TTL-bounded at 90 days. No compression or capacity concern at current fleet scale. </details> <!-- AI-Sessions dir: ~/.local/share/opencode/ (project: -Users-michaelusachenko-Desktop-Code-current-gkg) opencode session (2026-08-04) -->