Respond with 401 if no packages found in group with public registries

Context

In Allow anyone to pull public NuGet packages on g... (!155119 - merged), we tried to enable the feature that allow anyone to download NuGet packages from public registries, even if the parent group is private.

To do so, we introduced a new permission; that permission will be granted if the group has any public registries, regardless of its access level. However, this broke how NuGet handles the requests for private registries.

Mainly, NuGet client sends an anonymous request to the registry (GitLab's NuGet registry in this case), and if the registry responded with 401, NuGet attaches the token to the headers of the next request, so the request can be authenticated & authorized.

After enabling the allow_anyone_to_pull_public_nuget_packages_on_group_level feature flag, the needed 401 response wasn't sent in case if the group has any public registry. In this case, the new permission read_package_within_public_registries (introduced in !155119 (merged)) would be granted, and the anonymous request will pass the authorization phase and enter the finder.

The finder will look for the requested package that its project is public, or its registry is public. But if the requested package is in a private project/registry, the finder will return an empty result, and a 404 response will be returned to NuGet client. Which means NuGet will not be able to send the needed credentials in a subsequent request since it didn't receive the needed 401 from the NuGet registry.

What does this MR do and why?

Return 401 unauthenticated response if no packages were found in a group that has public registries. Let's break down the flow:

• When we execute nuget install package_name command, NuGet client sends an anonymous request to the NuGet Repository.
• The NuGet Repository receives the anonymous request and checks if the group has any public registries. If there are any, the anonymous request is granted the permission read_package_within_public_registries and proceed to the finder.
• The finder will search for the requested package in any public project or public registry in the group. If the package was found, we return it.
• If the package wasn't found, then we respond with 401 instead of 404. Why? Because it's an anonymous request, and we don't know if it's anonymous on purpose, or it's the normal anonymous request sent by default from NuGet client.
• If the request is anonymous on purpose, then the 401 response is better than 404 because it doesn't disclose any info about the existence of the requested package.
• If the request is anonymous because NuGet client expects a 401 so that it can send the token back with the subsequent request, then we avoid the bug that caused this incident by returning the correct status code (401 not 404).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

1. Make sure you have 2 private projects in a private group.

2. Open rails console:

   # Enable the ~"feature flag"
   Feature.enable(:allow_anyone_to_pull_public_nuget_packages_on_group_level)
   
   # Enable `package_registry_allow_anyone_to_pull_option` application setting
   ApplicationSetting.last.update(package_registry_allow_anyone_to_pull_option: true)
   
   # For one of the 2 private projects, enable Allow anyone to pull from Package Registry
   Project.find(<project1_id>).project_feature.update(package_registry_access_level: ::ProjectFeature::PUBLIC)
   
   # stub file upload
   def fixture_file_upload(*args, **kwargs)
     Rack::Test::UploadedFile.new(*args, **kwargs)
   end
   
   # Create a nuget package in the other private project (not the one with the public package registry)
   package = FactoryBot.create(:nuget_package, project_id: <project2_id>, package_name: 'hello.nuget')

3. We can now try installing the package using NuGet CLI:

   nuget install hello.nuget -OutputDirectory <output_directory> -Source "http://gdk.test:3000/api/v4/groups/<project2_id>/-/packages/nuget/index.json"

   nuget install will show a prompt asking for username and password, which means we correctly respond with 401 and that's why NuGet client tries to get the credentials to send them back with a subsequent request.

On master

• Clear NuGet cache and repeat step 3. => 404 response (wrong response)

Related to #471326