API endpoints captured as unauthenticated by RackAttack

Summary

1. Rack Attack logging as throttle_unauthenticated

2. Rails logging as a known user

In particular, the same basic endpoint/controller is reporting an authenticated user in the Rails logs, but RackAttack has seen it as unauthenticated. The problem endpoints are in the dependency proxy:

• Groups::DependencyProxyForContainersController#blob
• Groups::DependencyProxyForContainersController#manifest

Example Project

Example project can be found on https://gitlab.zendesk.com/agent/tickets/230844

What is the current bug behavior?

The customer gets a 429 rate limit errors for unauthenticated requests when they are actually authenticated using CI/CD with dependency proxy and performing docker pull from a self-managed runner that is connected to GitLab.com

What is the expected correct behavior?

The customer should not get 429 rate limit errors when request are authenticated until they hit the rate limit of 2000 request per minute

Relevant logs and/or screenshots

• https://gitlab.slack.com/archives/CB3LSMEJV/p1629323305299800

https://log.gprd.gitlab.net/goto/5616be443f6ded2bb326b3922d79ae69 shows the Rack_Attack logs for the instance they reported from yesterday (a little over 32 hours ago). One key fact is that there is no username and json.matched is throttle_unauthenticated, so this was an unauth per-IP rate-limit against $REDACTED_IP

And https://log.gprd.gitlab.net/goto/7088596e29af777ff2f1a9e330ae579c shows logs from that IP at that time, where we can see for 13:57 they did indeed exceed 500/min (the limit for unauth'd requests : https://docs.gitlab.com/ee/user/gitlab_com/#gitlabcom-specific-rate-limits).BUT: if we look for access logs from that IP without a username (https://log.gprd.gitlab.net/goto/a42ae8238f545ad4668d3b94833a5621), we only find 6.

So: the requests were authenticated, but RackAttack thought they weren't. We've seen this sort of behavior on other endpoints (back when we were enabling rate-limiting in the first place). It is likely a bug/weirdness in the auth handling

• https://log.gprd.gitlab.net/goto/5616be443f6ded2bb326b3922d79ae69
• https://log.gprd.gitlab.net/goto/7088596e29af777ff2f1a9e330ae579c
• https://log.gprd.gitlab.net/goto/a42ae8238f545ad4668d3b94833a5621
• https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/753
  • Previous similar situation; same basic symptoms, suggests a path for investivgation

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info

GitLab 14.1

Possible fixes