Skip to content

Geo secondary getting rate limited due to unauthenticated calls

Summary

We are seeing failures on Geo secondary because of being rate limited by the primary instance as unauthenticated. We don't expect this to happen since calls to primary should be authenticated and thus not face any rate limits

Steps to reproduce

  1. Ensure Geo's concurrency settings are high
  2. Ensure Sidekiq concurrency is high
  3. Have a lot of data
  4. Click Resync

Example Project

What is the current bug behavior?

Gitaly on secondary fails because of:

fetch remote: "remote: Request rate limited - please contact your GitLab admins\nfatal: unable to access 'https://dedicated-tenant.com/xxx/yyy/zzz.git/': The requested URL returned error: 429\n": exit status 128

On the primary webservice we see:

...
matched: throttle_unauthenticated_git_http
message: Rack_Attack
status: 429
severity: error
...

What is the expected correct behavior?

Geo Secondary should not be making any unauthenticated pulls from primary and thus not get rate limited

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

GitLab Dedicated v18.0.3 internal_reference: damp_grey_squirrel

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true)

(For installations from source run and paste the output of: sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)

(we will only investigate if the tests are passing)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by 🤖 GitLab Bot 🤖