### Problem to Solve The 0.131.0 rollout (schema v99 to v100, full rebuild) took 4 h 04 min to promote in production. During all of this time, users read v99 data, and v99 stopped at T0. So at promotion, the served data was 4 h old. Most of the time was a tail: 95% of namespaces were done after about 1 h 45 min, and the last 6 large namespaces took 2 h 20 min more. The tail was slow because failed full-backfill requests come back only on the hourly sweep, and the sweep loses to CDC requests on the same subject. The rollout also showed other gaps: a circuit breaker that is not wired, gate telemetry that fails at prod scale, and one namespace that promoted with no data. ### Proposed Solution - Keep served data fresh during a full rebuild (continue incremental writes to the active version, or dual-write). - Stop the starvation of backfill requests: separate the backfill subject from CDC, or merge targets on publish. - Retry incomplete namespaces within minutes during a migration, not on the hourly sweep. - Make a long namespace backfill survive interruptions: heartbeat before the first page, or one message per plan. - Fix the smaller defects in the list below (circuit breaker, KEDA tail capacity, gate telemetry, analytics limit, organization transfer). <details> <summary><b>Agent context</b>: rollout timeline, evidence, and code pointers</summary> Rollout: [apps!3852](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3852), 0.131.0, v99 to v100, `scope "*"`. All times UTC, 2026-09-28. #### Timeline | Time | Event | |---|---| | 14:02:26 | apps!3852 merged | | 14:05:34 | T0 prod: v100 `migrating`, 40 tables created | | 14:14:01 | Staging promoted v100 (T+8 min, 38 namespaces) | | 14:25-14:40 | Graph ClickHouse read degradation (see item 9) | | 15:06 | Sweep: 2,715 dispatched, 702 skipped | | 15:50 | Gate 3212/3419 (94%); tail starts | | 16:06 | Sweep: 3,190 dispatched, 228 skipped | | 16:54 | Code backfill drained: 1,086,112 of 1,093,442 v99 code checkpoints (99.3%) | | 17:06 | Sweep: 3,412 dispatched, 8 skipped; gate 3367 to 3405 | | 18:07 | Sweep: 3,423 dispatched, 0 skipped | | 18:10:02 | Prod promoted v100 (T+4 h 04 min), gate 3423/3423 | #### 1. Served data is frozen during a full rebuild After T0 the indexer writes only to `v100_*`. The last write to `v99_gl_job` was 14:04:47 and to `v99_gl_merge_request` 14:04:59, while `v100_*` had writes at 16:33. The webserver serves the active version (v99) until promotion. So the length of the backfill tail is the age of the data that users see. This makes every item below more expensive. #### 2. Sweep requests lose to CDC requests on the same subject - The full backfill request and CDC requests publish to the same subject, `sdlc.namespace.indexing.requested.1.<root>`. - The stream allows one message per subject (discard new). When a CDC message is queued or in flight, the sweep publish fails with `PublishDuplicate` and is counted as skipped: `crates/indexer/src/orchestrator/dispatch/namespace_indexing.rs:70`. - CDC requests carry only the changed targets (for example `targets=[Job]`). The SDLC handler skips every plan whose target is not selected: `crates/indexer/src/modules/sdlc/handler/entity.rs` (`indexing_requested`). - Busy namespaces have the most CDC traffic, so they lose the most often. These are also the largest namespaces, which need the full request most. - Example: namespace 131121267 lost its full request at 14:38 (max_deliver). Later runs were CDC-only, and it stayed at 1 of 46 plans until a sweep got through. #### 3. Recovery takes up to one hour - The full request used all 5 deliveries for many large namespaces. The `max_deliveries` reconciler deletes the message (`crates/indexer/src/orchestrator/max_deliveries.rs`), which is correct, but the only path back is the namespace sweep. - `sweep_interval_secs` is 3600 (`crates/orbit-server-config/src/engine.rs`). So each lost request costs up to 1 h of tail. The gate moved in steps at the :06 sweeps (see timeline). #### 4. One message runs all 46 plans in series - One namespace message runs 46 handlers in sequence. For the largest roots this is hours of work in one message. `Finding` alone was 200-420 M rows per root at about 2 M rows/min. - The SDLC pipeline sends a progress heartbeat only when it saves a page (`crates/indexer/src/modules/sdlc/pipeline.rs:275`). A slow query before the first page can exceed `ack_wait`, so JetStream redelivers while the first run continues. Probable, not proven. - Evidence of repeated runs: 89005436 reached attempt 3 on `ns.89005436.Finding`, and raw `v100_gl_finding` rows for that root were 146% of v99 before merges. For 131121267, 5 deliveries took 32 min (about 6 min each). - 89005436 finished `Finding` and then stopped with 20 plans missing until the 18:07 sweep. Root cause not confirmed. #### 5. KEDA gives the tail minimum capacity The SDLC ScaledObject scales on pending messages. In the tail, all work is in flight and the queue is empty (`0/100`), so the pool stayed at 6 pods (96/96 SDLC permits in use) while `maxReplicas` is 12. Scale-in from 12 to 6 around 15:45 also killed in-flight runs, which caused redeliveries. #### 6. GitLab client circuit breaker is not wired `CircuitBreakingGitlabClient` (`crates/gitlab-client/src/circuit_breaking.rs`) has tests but no production caller. The code indexer builds a raw `GitlabClient` (`crates/indexer/src/modules/code/mod.rs:68`). Prod had no `circuit` metrics or `CircuitOpen` errors while project-info 503s peaked at about 400/min. Note: a per-process breaker with the defaults (5 failures in 30 s, 60 s open) would have blocked all GitLab calls, including calls for healthy shards. A per-shard or per-project backoff fits this failure better. #### 7. Project-info 503s come from a hot Gitaly node - Rails `GET /api/:version/internal/orbit/project/:project_id/info` went from 10-22 req/min to about 11,400 req/min during the backfill. Errors peaked at 265/min at 15:11. Apdex stayed at 0.999. - The endpoint calls `project.repository.commit(default_branch_ref)` (Gitaly `FindCommit`). Rails maps `ResourceExhausted` to 503. - `gitaly-04-stor-gprd` (gitlab-gitaly-gprd-6b68) was at 95-98% CPU from 12:13, before T0. It produced 864 of about 1,000 fleet `FindCommit` `ResourceExhausted`/min. Most CPU was clone/fetch traffic (`PackObjectsHook`, `PostUploadPack`); `FindCommit` was about 1.1 of 19 CPU-s/s. - Failures repeat per project (repositories on that shard), so those projects go to the DLQ after 5 attempts (765 at peak) and wait for a code backfill re-dispatch. #### 8. Gate code-coverage telemetry fails at prod scale Every completion tick in prod logged `code coverage telemetry unavailable this tick ... count code-indexed projects: query error: invalid params: uri too long`. Staging (484 projects) reported coverage correctly. Promotion does not depend on it, but we had no code coverage signal in the gate log. #### 9. Backfill writes degrade graph reads `system.query_log` on one replica, 5 min buckets: | From | Writer SELECT p95 | Arrow insert p95 | Reader p95 | Reader timeouts (159) | |---|---|---|---|---| | 14:15 | 10 ms | 3.6 s | 4.5 s | 0 | | 14:25 | 2.9 s | 12.6 s | 16.9 s | 3 | | 14:30 | 4.2 s | 13.1 s | 22.6 s | 3 | | 14:45 | - | - | 6.3 s | 0 | No 202/439/252 errors. Webserver code 159 errors: 1 in 12:00-14:00, 27 in 14:00-15:00. It recovered without action as the code pool throttled itself. #### 10. Analytics events dropped for long entity names `failed to build indexing analytics event: entity_type longer than 64 characters` for edge plans such as `HAS_VULNERABILITY_siphon_sbom_occurrences_vulnerabilities_Vulnerability`. Indexing is not affected; the events are lost. #### 11. Organization transfer indexes a namespace as empty - Root 97155062 moved from organization 1 to 3000449 at 12:17 (before T0). The v100 group row has `traversal_path` `3000449/97155062/`. - v100 marked all 46 plans for `ns.97155062` as indexed (`indexed_at` 18:07:56) with 0 rows: 0 projects (v99: 105 ids), 0 MRs (v99: 508), 0 vulnerabilities. No project was found under any path in v100. - The gate counted it as complete. This is the empty-checkpoint class of #794, from a different cause. #### Completeness check (for reference) Deduplicated live ids, v100 / v99: project 1.007, group 1.013, user 0.995, runner 1.003, container_repository 1.017, branch 0.996. Raw `total_rows` drops (for example `gl_project` 0.76) were unmerged duplicates in the cloned v99 tables. `gl_vulnerability` live 0.888: 11.16 M of the gap is root 188131, which has no v100 checkpoints (not enabled; last v99 write 2026-09-24), so expected. #### Suggested split 1. Fresh served data during a rebuild (item 1). 2. Backfill request starvation and retry cadence (items 2, 3). 3. Long namespace runs: heartbeat and plan granularity (item 4). 4. KEDA capacity for in-flight work (item 5). 5. GitLab client backoff (items 6, 7). 6. Small fixes: gate telemetry URI (8), analytics name limit (10). 7. Organization transfer (11), probably `orbit::indexing`. </details>