Skip to content

NuGet package source API rate limits

Summary

We have published a few nuget packages in the package registry (gitlab.com). We also have two runners that take care of our pipeline (not using the shared runners of gitlab).

When restoring the nuget packages, nuget asks all sources for all required packages. So the gitlab registry will also be called for packages that are on nuget.org.

With the rate limits used on gitlab.com (see docs and current settings) our runners constantly get 429 Too many requests as soon as a few .net jobs are running.

This basically renders the gitlab package repository completly useless for us. Nuget is a pretty request intensive package manager and the current rate limits of gitlab.com seem way too low.

Steps to reproduce

  • Pipeline building a .NET project, preferrably something with a bunch of references like an AspNet Core API project
  • Add a nuget package source using a package registry on gitlab.com
  • Pipeline should run on an own runner (you probably won't get the error on shared runners)
  • Start multiple (3-4?) build jobs (dotnet restore should be enough)

What is the current bug behavior?

Constantly getting 429 Too many requests for the NuGet package registry API

What is the expected correct behavior?

Higher rate limits for the NuGet package registry API, so that a normal usage doesn't constantly run against the rate limits.

Relevant logs and/or screenshots

Microsoft (R) Build Engine version 16.7.0-preview-20360-03+188921e2f for .NET
Copyright (C) Microsoft Corporation. All rights reserved.
  Determining projects to restore...
  Retrying 'FindPackagesByIdAsync' for source 'https://gitlab.com/api/v4/projects/<redacted>/packages/nuget/download/system.resources.resourcemanager/index.json'.
  Response status code does not indicate success: 429 (Too Many Requests).
  Retrying 'FindPackagesByIdAsync' for source 'https://gitlab.com/api/v4/projects/<redacted>/packages/nuget/download/system.reflection.extensions/index.json'.
  Response status code does not indicate success: 429 (Too Many Requests).
  Retrying 'FindPackagesByIdAsync' for source 'https://gitlab.com/api/v4/projects/<redacted>/packages/nuget/download/hangfire.sqlserver/index.json'.
  Response status code does not indicate success: 429 (Too Many Requests).
  Retrying 'FindPackagesByIdAsync' for source 'https://gitlab.com/api/v4/projects/<redacted>/packages/nuget/download/microsoft.extensions.localization.abstractions/index.json'.
  Response status code does not indicate success: 429 (Too Many Requests).
[...]

Output of checks

This bug happens on GitLab.com

Possible fixes

Probably somewhere around here: https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/blob/master/attributes/default.rb#L66