Nuget (group) package registry has high CPU when restoring large projects

We're seeing an issue on our self hosted gitlab & nuget package registry, where our gitlab CPU usage spikes when we do restores on large-ish projects (100-200 csproj's in one solution). The packages we'll be restoring will be a few hundred of our own packages, plus hundreds of packages from Nuget.org. When this happens, especially when we push new versions of our own packages and thus no local endpoint will have a cache, we see loads like this.

image

I've been checking some logs, and it seems like a number of things go wrong:

  • Nuget will do an unlimited number of connections at once, this is rough behavior, but there isn't an easy way to mass-configure clients to not overload the gitlab server with connections.
  • Nuget will ask all sources, for all packages
    • The nuget client has no way of knowing which source has which packages, so due to our ordering, Gitlab will receive requests for all packages (as we want our packages instead of official packages)
    • This means Gitlab needs to have a robust and fast negative-cache for Nuget packages, to ensure all these "bogus" requests don't overload anything
  • Gitlab does not redirect package downloads to GCS (#370103 (closed))

I'm hoping something can be done to speed up Gitlabs nuget package registry. It's quite slow at this point, and we're not even storing that many packages in it (some 50k package versions, across ~160 packages - or about 300 versions of each package). I have an inkling feeling that the requests for packages that don't exist, is what really slows everything down.

Is there a fast negative-cache of non-existent packages in gitlab?

This issue becomes worse, as more sources are added. For some projects, we have two sources on gitlab, one pointing to a group and another to a prototype project not in any group. In this case, the client will ask Gitlab, twice for all packages, as there are now two unique sources.

Edited by Michael Bisbjerg