[SPIKE] Investigate Kibana logs for ref_exists_in_gitaly? fallback
## Context In version %"18.0", we introduced https://gitlab.com/gitlab-org/gitlab/-/merge_requests/188492+, which serves as a fallback when the Redis cache is stale and the reference exists in Gitaly. As of today, we have [3,452 discrepancies](https://log.gprd.gitlab.net/app/r/s/iwNRd) between the Redis cache and Gitaly. ## Expectations - Investigate these cases and track the different services using the correlation ID in Kibana. - Try to identify a common pattern or potential causes by reviewing the logs. - Try to solve the following questions: Why are we using the fallback? What makes these requests different? - If a common pattern or root cause is not found. What information are we missing? Do we need to add more logging? ## References - https://gitlab.com/gitlab-org/gitlab/-/issues/536824+ - https://gitlab.com/gitlab-org/gitlab/-/issues/536824#note_2507976887 - https://gitlab.com/gitlab-org/gitlab/-/issues/536824#note_2512117608
issue