## Description Over the past 30 days, we've identified that 79% of infrastructure-related pipeline job failures are due to the error pattern `"fatal: couldn't find remote ref"`. This is our most frequent git-related failure and impacts developer productivity significantly. ## Affected Jobs 164 jobs total out of 206 analyzed jobs (79% of infrastructure failures) ## Error Logs Example error logs from one of a job ``` Fetching changes with git depth set to 20... Initialized empty> Git repository in /builds/gitlab-org/gitlab/.git/ Created fresh repository. fatal: couldn't find remote ref refs/pipelines/1821944447 Getting source from Git repository Fetching changes with git depth set to 20... Initialized empty Git repository in /builds/gitlab-org/gitlab/.git/ Created fresh repository. fatal: couldn't find remote ref refs/pipelines/1821944447 Uploading artifacts for failed job Uploading artifacts... WARNING: coverage/: no matching files. Ensure that the artifact path is relative to the working directory (/builds/gitlab-org/gitlab) ERROR: No files to upload Cleaning up project directory and file based variables ERROR: Job failed: exit code 1 ``` ## Preliminary Analysis This error occurs when GitLab CI runners cannot locate the pipeline-specific git references (refs/pipelines/<ID>) used to maintain pipeline stability. Correct Pipeline ID exists :white_check_mark: And on restart, the job works :thinking: , for example | Failed Job | Succeessful job | | ------ | ------ | | https://gitlab.com/gitlab-org/gitlab/-/jobs/10062980422 | https://gitlab.com/gitlab-org/gitlab/-/jobs/10072905834 | | | | This error pattern accounts for the vast majority of our git-related failures, suggesting it should be our highest priority. ## Next Steps - Engage with git domain experts / teams - Implement better retry mechanisms for pipelines that encounter this error - Consider a fallback mechanism