Handling various token formats in the package API

Problem

As we expand GitLab's package registry offering to more and more package management systems, we increasingly find that each system has a different way of handling authentication and credentials.

Users have requested a variety of ways to authenticate with these package managers: Personal Access Tokens, Deploy Tokens, and Job Tokens.

Each of these token types are set up with a different use case in mind, and generally use their own unique headers, which allows the auth flow to understand which authentication method and credentials it is working with. The problem is that package managers are often not configurable to include custom headers. For example, NPM will send all requests with a Authorization: Bearer <token> (Oauth) header.

This causes a conflict between keeping tokens for separate functions while allowing users to use all tokens in the same way (with the same headers).

In terms of code, in the API we see

      def find_user_from_sources
        find_user_from_access_token ||
          find_user_from_job_token ||
          find_user_from_warden
      end

First we check to see if a PAT is present, then check for a job token, and last we check for warden auth. In the case of NPM, however, we are passing a PAT in without the standard PAT headers, so we needed to update find_user_from_access_token to also check for Oauth formatting, which is a bit hacky. Then for other package managers, we have overwritten methods that occur in auth_finders.rb (see here and here for examples). This creates complexity that contributes to difficulty in tracing the auth flow for any given package manager, and in the case that we want to add a new auth method (deploy tokens for example), it is difficult to understand each type of auth that needs to be updated and where.

Proposal

There is no way to avoid having to deal with these varying auth workflows, so I think it would be best if we at least were able to group all of the overrides and additional logic in one module. I think this is starting to come together already in API::Helpers::Packages with BasicAuthHelpers, but there is still other logic that lives in ConanToken, Conan::ApiHelpers, and in Gitlab::Auth::AuthFinders. The last example has an especially difficult to see npm-specific functionality where (https://gitlab.com/gitlab-org/gitlab/-/blob/ad2cc99686f75260f11cde8edab9b1f7d4baf254/lib/gitlab/auth/auth_finders.rb#L148) is where the PAT is checked with OAuth headers.

Outside of bringing this logic into a common namespace, I think it's worth discussing to see if there are maybe other ideas of how this could be organized in a discoverable and expandable way.

Additional notes

Current understanding of the auth for each package manager:

Package Manager Auth setup
NPM Oauth format headers Authorization: Bearer <token>
Maven Customizeable headers in settings.xml, docs currently use Private-Token header, we could potentially use Basic auth headers here to allow for username and password?
Conan Authorization request is made with basic auth, then GitLab returns an encoded JWT that is used for subsequent requests. This could be modified to handle deploy tokens
Nuget Basic auth - un/pw
PyPI Basic auth - un/pw
GoProxy .netrc file that uses basic auth headers
Composer Oauth format headers Authorization: Bearer <token>

References

Edited by Giorgenes Gelatti