npm package API rate limits

Summary

There is an issue when pulling npm packages using GitLab CI. Users are reporting that they are hitting a rate limit and having the request rejected. Self-Managed instances can increaes this limit in their HAProxy settings by adjusting rate_limit_http_rate_per_minute and rate_limit_sessions_per_second. But for GitLab.com, the HAProxy responds with an HTTP status code 429 to API requests that exceed 10 requests per second per IP address.

User feedback

• The issue was originally reported when someone tweeted about an error interacting with the npm registry
• "Our project uses npm-scope and it should only fetch a few npm packages from the private repository. But we get the error "E429 - Too Many Requests - GET https://gitlab.com/api/v4/packages/npm/is-absolute-url" for packages which should be fetched from the public repository."

Error

npm ERR! 429 Too Many Requests - GET https://gitlab.com/api/v4/packages/npm/chardet

Steps to reproduce

Example Project

What is the current bug behavior?

Rate limiting occurs

What is the expected correct behavior?

If rate limiting is expected behaviour, then it's documented somewhere like

• https://docs.gitlab.com/ee/security/rate_limits.html
• https://docs.gitlab.com/ee/administration/instance_limits.html#package-registry-limits

And some guidance on why it's occurring might also be needed.

Relevant logs and/or screenshots

From job log

Getting source from Git repository
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes...
Initialized empty Git repository in /builds/masksfordocs/toolbox/.git/
Created fresh repository.
Checking out 4ecd9e88 as pedro-gutierrez-okteto...
Skipping Git submodules setup
Restoring cache
Checking cache for ui-17641289-2...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/17641289/ui-17641289-2 
Successfully extracted cache
Downloading artifacts
Downloading artifacts for build_ui (716156553)...
Downloading artifacts from coordinator... ok        id=716156553 responseStatus=200 OK token=
Executing "step_script" stage of the job script
$ cd ${REACT_DIR}
$ npm config set registry https://gitlab.com/api/v4/packages/npm/
$ npm ci
npm WARN prepare removing existing node_modules/ before installation
npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://gitlab.com/api/v4/packages/npm/chardet
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-09-02T14_43_57_064Z-debug.log
ERROR: Job failed: exit code 1

Output of checks

This bug happens on GitLab.com

Job failed 24 hours prior to raising this issue. At the time of raising the issue, GitLab.com was: 13.4.0-pre 5c96eb9a

Possible fixes