## Summary When a push (or merge commit) creates a pipeline, a **transient** Gitaly "Commit not found" during pipeline creation causes the pipeline to be **silently dropped** — no pipeline object is created, no user-facing error, and **Sidekiq does not retry**. The same underlying transient miss, when it surfaces through a different code path, *does* self-heal via Sidekiq retry. This asymmetry means pipeline creation is non-deterministically dropped or healed depending on which Gitaly lookup trips the inconsistency window. Observed on a self-managed instance where Gitaly repository storage is on shared NAS (multi-node, read-after-write consistency gap between nodes), but the robustness gap described here is storage-agnostic — it applies to any transient Gitaly unavailability during pipeline creation. Originally investigated in [RFH gitlab-com/request-for-help#4697](https://gitlab.com/gitlab-com/request-for-help/-/work_items/4697); see the [root-cause analysis note](https://gitlab.com/gitlab-com/request-for-help/-/work_items/4697#note_3657889564). ## Steps to reproduce (the mechanism) `Gitlab::Ci::Pipeline::Chain::Build#perform!` reads two ref-derived attributes sequentially: ```ruby @pipeline.assign_attributes( ... sha: @command.sha, # read first ... tag: @command.tag?, # read second ... ) ``` Both resolve the ref via Gitaly, but they handle a transient miss **inconsistently**: - **`command.sha`** → `Project#commit` → `Gitlab::Git::Commit.find` **swallows** `Gitlab::Git::CommandError` (and `NoRepository` / `ArgumentError`) and returns `nil`. A missing/invisible commit can also come back as an empty-but-`OK` `FindCommit` response (no exception at all). Either way, `sha` is `nil`. - **`command.tag?`** → `RefResolver` → `Repository#ref_exists?` → `wrapped_gitaly_errors` **re-raises** as `Gitlab::Git::CommandError` (no swallowing rescue in that path). So during a read-after-write window: **Scenario 1 — silent drop (the bug):** 1. Commit not yet visible → `command.sha` returns `nil` (swallowed). 2. Commit becomes visible → `command.tag?` succeeds. 3. `Chain::Validate::Repository` sees `sha` nil → emits the **non-raising** `error('Commit not found')`. 4. `CreatePipelineWorker` logs `WARN "Error creating pipeline"` with `retry: 0`, finishes `job_status: done` → **no retry, no pipeline**. **Scenario 2 — self-heal:** 1. Commit not yet visible → `command.sha` returns `nil`. 2. Commit still not visible → `command.tag?` **raises** `Gitlab::Git::CommandError`. 3. Job **fails** → Sidekiq `retry: 3` requeues → succeeds on a later attempt → pipeline created. ## Local reproduction Reproduced in GDK by simulating the transient miss in `Command#sha` for two test branches: - `repro-drop` → `sha` returns `nil` → **Scenario 1** → `GET /pipelines?sha=<merge_commit>` returns `[]`, stays empty, no retry. - `repro-heal` → `sha` raises `Gitlab::Git::CommandError` once → **Scenario 2** → job fails, Sidekiq retries, push pipeline created on the retry. ### Scenario 1 log (`repro-drop`) — silent drop ```json {"severity":"WARN","time":"...","class":"CreatePipelineWorker","project_path":"root/repro-race","message":"Error creating pipeline","errors":"Commit not found","pipeline_params":{"before":"34a65a8a…","after":"d37b1e0f…","checkout_sha":"d37b1e0f…","ref":"refs/heads/repro-drop"},"retry":0} {"severity":"INFO","time":"...","class":"CreatePipelineWorker","message":"CreatePipelineWorker JID-…: done","job_status":"done"} ``` Note `before` is a **real parent SHA** (not the zero-SHA), and the worker ends `job_status: done` — Sidekiq never retries. ### Scenario 2 log (`repro-heal`) — self-heal via retry ```json {"severity":"WARN","time":"...","class":"CreatePipelineWorker","retry_count":1,"exception.class":"Gitlab::Git::CommandError","exception.message":"simulated transient read-after-write (exit status 128)","message":"CreatePipelineWorker JID-…: fail","job_status":"fail"} {"severity":"INFO","time":"...","class":"CreatePipelineWorker","retry_count":2,"message":"CreatePipelineWorker JID-…: start","job_status":"start"} {"severity":"INFO","time":"...","class":"Ci::InitialPipelineProcessWorker","args":["2029"],"job_status":"start"} {"severity":"INFO","time":"...","class":"CreatePipelineWorker","retry_count":2,"message":"CreatePipelineWorker JID-…: done","job_status":"done"} ``` Same `jid` across attempts, `retry_count` 1 → 2, push pipeline created on the second attempt. ## What is the current bug behavior? A transient Gitaly "Commit not found" during pipeline creation is treated as a permanent validation failure → the pipeline is silently dropped with no retry and no user-facing error. ## What is the expected correct behavior? A transient "Commit not found" should be retried (like the `ref_exists?` path already is), so the pipeline self-heals once the commit is consistently visible — instead of being dropped. ## Relevant code - `lib/gitlab/ci/pipeline/chain/build.rb` — `perform!` reads `sha` then `tag?`. - `lib/gitlab/ci/pipeline/chain/command.rb` — `#sha` (swallows via `Commit.find`), `#tag?` / `#ref_exists?` (raises). - `lib/gitlab/ci/pipeline/chain/validate/repository.rb` — `error('Commit not found')` (non-raising validation error). - `lib/gitlab/git/commit.rb` — `find` rescues `CommandError`/`NoRepository`/`ArgumentError` → `nil`. - `app/workers/create_pipeline_worker.rb` — `raise_reference_not_found_error!` only retries on `"Reference not found"` + blank `before`; `sidekiq_options retry: 3`. ## Related - Related to the Praefect replication-lag retry added in !235792 / #504460, which only covers `"Reference not found"` on new-ref pushes (blank `before`). The merge-commit case here is `"Commit not found"` with a real `before` SHA, so it falls outside that guard. - Upstream read-after-write context: [gitaly#4255](https://gitlab.com/gitlab-org/gitaly/-/issues/4255), [gitaly#6759](https://gitlab.com/gitlab-org/gitaly/-/issues/6759). ## Proposal (outline) 1. Promote `'Commit not found'` to a named constant in `Validate::Repository`. 2. Extend `CreatePipelineWorker#raise_reference_not_found_error!` to also raise `ReferenceNotFoundError` on `"Commit not found"` (reusing the existing retry + `retry: 3` machinery), relaxing the `blank_ref?(before)` guard for this case. 3. Gate behind a feature flag (sibling of `ci_create_pipeline_worker_retry_on_reference_not_found`). 4. Add regression specs. ## Open design decisions 1. **The `blank_ref?(before)` guard.** For `"Commit not found"` the retry must fire regardless of `before` (merge commits have a real parent SHA, not the zero-SHA). Relaxing it widens what gets retried: - **Concern:** a genuinely-invalid SHA (real user error) would now retry 3× before giving up. - **Option (a):** accept it — `retry: 3` bounds the cost, and the failure is still eventually logged. - **Option (b):** scope the retry to the transient signature — only retry when `ref_exists?` is true **but** `sha` is nil. Note these two check *different* things via *different* Gitaly RPCs: - `ref_exists?` → `resolved_ref` → `RefExists` on the **ref name** (e.g. `refs/heads/<branch>`). - `sha` → `project.commit(origin_sha || resolved_ref)` → `FindCommit` on a **commit SHA**, where `origin_sha` = `checkout_sha || after_sha` (for a push/merge, `after_sha` is the just-written commit). So `ref_exists? == true && sha.nil?` means *"the branch pointer resolves, but the just-written commit it points at isn't yet readable on this Gitaly node"* — the exact read-after-write signature. This is almost always transient (the commit demonstrably exists, since the ref points at it), whereas a genuinely-invalid user SHA fails differently. More surgical than (a); avoids retrying legitimately-missing commits. 2. **Where the retry decision lives.** Keep it in the worker (matches the existing `raise_reference_not_found_error!` pattern), or give `"Commit not found"` a dedicated non-persistable `failure_reason` in the chain and retry on that (heavier: touches the enum + presenter). Worker-level is the minimal change.