<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=504460) </details> <!--IssueSummary end--> <!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "type::bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary <!-- Summarize the bug encountered concisely. --> This issue has been created after a customer reported this behaviour, and we confirmed it was happening on GitLab.com (based on Elasticsearch Sidekiq logs). When a branch gets created, this results in a `POST /api/v4/internal/post_receive` request to GitLab with the changes. GitLab processes these changes, which includes creating pipelines. The `Ci::CreatePipelineService` checks if the ref exists via `Gitlab::Ci::Pipeline::Chain::Validate::Repository`. If the branch doesn't exist, it returns `Reference not found`: https://gitlab.com/gitlab-org/gitlab/-/blob/94f870b8e4b965a41dd2ad576d50f7eeb271f117/lib/gitlab/ci/pipeline/chain/validate/repository.rb#L13 It seems like there might be two different ways to check if a branch exists: 1) Check the cached data, and 2) Call Gitaly using `Gitaly::RefExistsRequest` As mentioned previously, we can see logs for GitLab.com where `POST /api/v4/internal/post_receive` requests resulted in no pipeline created because `Reference not found`. That is despite the `pipeline_params.before` having the value `0000000000000000000000000000000000000000` for that ref. Could outdated cached branch data be the cause of this problem? Similar issue to #500751, which has been fixed. ### Steps to reproduce <!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. --> Unable to reproduce reliably, but customer see this intermittently, and we can see this occurs on GitLab.com ### Example Project <!-- If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report. If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version. --> ### What is the current *bug* behavior? After branch creation, pipelines can't run for that ref because `Reference not found`. This problem resolves itself over a period of time. <!-- Describe what actually happens. --> ### What is the expected *correct* behavior? After branch creation, pipelines can run for that ref, even those related to the `post_receive` request for that ref creation <!-- Describe what you should see instead. --> ### Relevant logs and/or screenshots <!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise. --> <details><summary>Elasticsearch query (Sidekiq index pattern)</summary> json.message:"Error creating pipeline" AND json.pipeline_params.before:"0000000000000000000000000000000000000000" AND json.errors:"Reference not found" </details> https://log.gprd.gitlab.net/app/r/s/uGAZV ### Output of checks <!-- If you are reporting a bug on GitLab.com, uncomment below --> <!-- This bug happens on GitLab.com --> <!-- and uncomment below if you have /label privileges --> <!-- /label ~"reproduced on GitLab.com" --> <!-- or follow up with an issue comment of `@gitlab-bot label ~"reproduced on GitLab.com"` if you do not --> #### Results of GitLab environment info <!-- Input any relevant GitLab environment information if needed. --> <details> <summary>Expand for output related to GitLab environment info</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`) </pre> </details> #### Results of GitLab application Check <!-- Input any relevant GitLab application check information if needed. --> <details> <summary>Expand for output related to the GitLab application check</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`) (we will only investigate if the tests are passing) </pre> </details> ### Possible fixes <!-- If you can, link to the line of code that might be responsible for the problem. --> <!-- If you don't have /label privileges, follow up with an issue comment of `@gitlab-bot label ~"type::bug"` -->