API calls return 429 from Cloudflare without reaching rate limits

Summary

Cloudflare says I'm rate-limited but RateLimit-* headers say the opposite.

I checked docs announcing 2000req/min. I never reached such high numbers, I get dropped after two or three hundreds of API calls.

Steps to reproduce

  1. call many (a few hundred) authenticated API calls (pipeline list in a project, job list in a pipeline), watch and record RateLimit-* headers
  2. occasionally, an HTTP error from CloudFront (with HTTP status 429) is returned on API
  3. The last successful request still has a high RateLimit-Remaining header value.

Requests come from the same IP using the same access token if that matters.

Example Project

N/A

What is the current bug behavior?

Rate-limiting is triggered much sooner than expected, after a few hundred calls in a minute (while RateLimit-Remaining reports approximately 1750)

What is the expected correct behavior?

2000 API calls per minute should be allowed as promised in the documentation.

Relevant logs and/or screenshots

Partial output from our tool that searches and downloads artifacts from recent pipelines. Note the timestamps and headers returned

2022-10-17 14:45:50,272 DEBUG https://gitlab.com:443 "GET /api/v4/projects/16539767/pipelines/668337420/jobs?scope=success&per_page=100 HTTP/1.1" 200 None
2022-10-17 14:45:50,279 DEBUG RateLimit-Observed='243' RateLimit-Remaining='1757' RateLimit-Reset='1666018010' RateLimit-ResetTime='Mon, 17 Oct 2022 14:46:50 GMT' RateLimit-Limit='2000'
2022-10-17 14:45:50,670 DEBUG https://gitlab.com:443 "GET /api/v4/projects/16539767/pipelines/668274720/jobs?scope=success&per_page=100 HTTP/1.1" 200 None
2022-10-17 14:45:50,671 DEBUG RateLimit-Observed='245' RateLimit-Remaining='1756' RateLimit-Reset='1666018010' RateLimit-ResetTime='Mon, 17 Oct 2022 14:46:50 GMT' RateLimit-Limit='2000'
2022-10-17 14:45:51,324 DEBUG https://gitlab.com:443 "GET /api/v4/projects/16539767/pipelines/668262296/jobs?scope=success&per_page=100 HTTP/1.1" 200 None
2022-10-17 14:45:51,333 DEBUG RateLimit-Observed='248' RateLimit-Remaining='1752' RateLimit-Reset='1666018011' RateLimit-ResetTime='Mon, 17 Oct 2022 14:46:51 GMT' RateLimit-Limit='2000'
2022-10-17 14:45:51,838 DEBUG https://gitlab.com:443 "GET /api/v4/projects/16539767/pipelines/668260654/jobs?scope=success&per_page=100 HTTP/1.1" 200 None
2022-10-17 14:45:51,838 DEBUG RateLimit-Observed='250' RateLimit-Remaining='1750' RateLimit-Reset='1666018011' RateLimit-ResetTime='Mon, 17 Oct 2022 14:46:51 GMT' RateLimit-Limit='2000'
2022-10-17 14:45:51,873 DEBUG https://gitlab.com:443 "GET /api/v4/projects/16539767/pipelines/668239798/jobs?scope=success&per_page=100 HTTP/1.1" 429 5357
2022-10-17 14:45:51,874 DEBUG 
2022-10-17 14:45:51,874 ERROR <!DOCTYPE HTML>
...
<title>Access denied | gitlab.com used Cloudflare to restrict access</title>
...
        <h2 class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light">You are being rate limited</h2>
...

The full logs available in private project: here or here

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

N/A

Results of GitLab application Check

N/A

Possible fixes

I don't know, cloudfront rate limits are possibly inconsistent with backend configuration.