Delay in updating open Merge Request and Issue count badges on Geo secondary

Summary

It takes around 20 minutes to update open Merge Request count on the left nav bar of a project on the Geo secondary.

Looks like the cache key open_merge_requests_count is not updated immediately after the merge request is replicated.

Workarounds:

• Run sudo gitlab-rake cache:clear on the Secondary. This clears cache at instance level.
• To just refer the open_merge_requests_count cache, run the following in Rails console:

p = Project.find(1) # Replace 1 with project ID 
cache = Projects::OpenMergeRequestsCountService.new(p)
cache.refresh_cache # refresh cache

Steps to reproduce

• Create a project
• Create a merge request
• Visit the project page on the Geo secondary site
• The badge on Merge Requests in the left nav bar is 0 but should be 1

What is the current bug behavior?

It takes 20 minutes to refresh/update the cache.

What is the expected correct behavior?

Cache should refresh/update immediately after the MR is replicated.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

GitLab 13.11.4-ee.

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes