Operations dashboard missing pipelines on Gitaly Cluster with multiple nodes via TCP when default branch is 'main'

Summary

The operations dashboard can fail to display data for valid pipelines, displaying The branch for this project has no active pipeline configuration instead. This happens when GitLab is using Gitaly Cluster with multiple Gitaly nodes and the instance default branch is set to main.

Discovered while trying to get QA tests to pass when the default branch is set to main: 9f56d7cd (comment 503534517)

The bug does not occur if there are multiple Gitaly nodes but they use TCP, or if there is only a single Gitaly node using sockets. It also doesn't occur if the default branch hasn't been changed (i.e., it still uses master).

Steps to reproduce

1. Configure GitLab to use Gitaly Cluster with at least 2 Gitaly nodes (the QA tests use 3. One Praefect node is enough) using TCP not unix sockets.
2. Create a few projects with pipelines (the QA test creates 4).
3. View the Operations Dashboard.

I was able to reproduce this on GDK with the following configuration:

1. Set node_count to 3 and then gdk reconfigure.

2. In gitaly/gitaly-0.praefect.toml, gitaly/gitaly-1.praefect.toml, and gitaly/gitaly-2.praefect.toml, listen on TCP instead of a socket:

   # gitaly/gitaly-0.praefect.toml
   # socket_path = "/home/mark/gdk/gitaly-praefect-0.socket"
   listen_addr = "192.168.20.9:9999"

   # gitaly/gitaly-1.praefect.toml
   # socket_path = "/home/mark/gdk/gitaly-praefect-1.socket"
   listen_addr = "192.168.20.9:9998"

   # gitaly/gitaly-2.praefect.toml
   # socket_path = "/home/mark/gdk/gitaly-praefect-2.socket"
   listen_addr = "192.168.20.9:9997"

3. In gitaly/praefect.config.toml listen on TCP instead of a socket and use TCP to reach the Gitaly nodes

   listen_addr = "192.168.20.9:2305"
   # socket_path = "/home/mark/gdk/praefect.socket"
   
   [[virtual_storage.node]]
     storage = "praefect-internal-0"
     address = "tcp://192.168.20.9:9999"
   
   [[virtual_storage.node]]
     storage = "praefect-internal-1"
     address = "tcp://192.168.20.9:9998"
   
   [[virtual_storage.node]]
     storage = "praefect-internal-2"
     address = "tcp://192.168.20.9:9997"

4. In gitlab/config/gitlab.yml use TCP to access the storage

   repositories:
     storages:
       default:
         path: /home/mark/gdk/repositories
         gitaly_address: tcp://192.168.20.9:2305

What is the current bug behavior?

The dashboard doesn't display all the expected project data. It only displays some of the results. Refreshing causes different results to be shown.

What is the expected correct behavior?

The dashboard should consistently display all available projects.

Relevant logs and/or screenshots