npm package API rate limits
Summary
There is an issue when pulling npm packages using GitLab CI. Users are reporting that they are hitting a rate limit and having the request rejected. Self-Managed instances can increaes this limit in their HAProxy settings by adjusting rate_limit_http_rate_per_minute and rate_limit_sessions_per_second. But for GitLab.com, the HAProxy responds with an HTTP status code 429 to API requests that exceed 10 requests per second per IP address.
User feedback
- The issue was originally reported when someone tweeted about an error interacting with the npm registry
- "Our project uses npm-scope and it should only fetch a few npm packages from the private repository. But we get the error "E429 - Too Many Requests - GET https://gitlab.com/api/v4/packages/npm/is-absolute-url" for packages which should be fetched from the public repository."
Error
npm ERR! 429 Too Many Requests - GET https://gitlab.com/api/v4/packages/npm/chardetSteps to reproduce
Example Project
What is the current bug behavior?
Rate limiting occurs
What is the expected correct behavior?
If rate limiting is expected behaviour, then it's documented somewhere like
- https://docs.gitlab.com/ee/security/rate_limits.html
- https://docs.gitlab.com/ee/administration/instance_limits.html#package-registry-limits
And some guidance on why it's occurring might also be needed.
Relevant logs and/or screenshots
Getting source from Git repository
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes...
Initialized empty Git repository in /builds/masksfordocs/toolbox/.git/
Created fresh repository.
Checking out 4ecd9e88 as pedro-gutierrez-okteto...
Skipping Git submodules setup
Restoring cache
Checking cache for ui-17641289-2...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/17641289/ui-17641289-2
Successfully extracted cache
Downloading artifacts
Downloading artifacts for build_ui (716156553)...
Downloading artifacts from coordinator... ok id=716156553 responseStatus=200 OK token=
Executing "step_script" stage of the job script
$ cd ${REACT_DIR}
$ npm config set registry https://gitlab.com/api/v4/packages/npm/
$ npm ci
npm WARN prepare removing existing node_modules/ before installation
npm ERR! code E429
npm ERR! 429 Too Many Requests - GET https://gitlab.com/api/v4/packages/npm/chardet
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2020-09-02T14_43_57_064Z-debug.log
ERROR: Job failed: exit code 1Output of checks
This bug happens on GitLab.com
Job failed 24 hours prior to raising this issue. At the time of raising the issue, GitLab.com was: 13.4.0-pre 5c96eb9a