GITLAB_THROTTLE_USER_ALLOWLIST still rate limit user

Overview

We can specify GITLAB_THROTTLE_USER_ALLOWLIST to allow specific users to bypass authenticated request rate limiting.

We've tried to set this up in https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24321#note_1545144216 for gitalyctl-staging user (id = 3378358) however as we can see below we still get the rate limit headers:

$ curl -I -s --header "PRIVATE-TOKEN: $(glsh vault show-secret k8s ops-gitlab-gke/gitalyctl/gstg | jq -r '.gitalyctl_api_token')" "https://staging.gitlab.com/api/v4/user" | rg 'rate'
ratelimit-observed: 1
ratelimit-remaining: 599
ratelimit-reset: 1694005282
ratelimit-resettime: Wed, 06 Sep 2023 13:01:22 GMT
ratelimit-limit: 600

At application start up time in auth.json we do see that user ID in the list so rails is loading it:

git@gitlab-webservice-web-7554964554-jk294:/$ cat /srv/gitlab/log/auth_json.log

{"severity":"INFO","time":"2023-09-06T12:40:24.243Z","gitlab_throttle_user_allowlist":[xxxxx,3378358]}

We also see it in the logs that the request from the user is in the throttle_allow_list

list of logs

source

What we've tried

Goal

When we add a user to GITLAB_THROTTLE_USER_ALLOWLIST it shouldn't get rate limited, nor should it get the rate limit headers. Same as when we allow list the IP.

Why

At the moment we can't remove rate limits on GitLab.com per user, but only by IP.

Edited by Steve Xuereb