Skip to content

Maven package registry is behaving inconsistently when using Group URL with 'Basic HTTP Authentication'

According to gitlab documentation maven package registry can be authenticated in two ways; "Basic HTTP Authentication" and "Custom HTTP Header". We have been using Basic HTTP Authentication for quiet some time now on a group URL. Our settings.xml looks like this;

        <server>
            <id>gitlab-saas-maven</id>
            <username>${env.GROUP_DEPLOY_TOKEN_NAME}</username>
            <password>${env.DEPLOY_TOKEN_VALUE}</password>

            <configuration>
                <authenticationInfo>
                    <userName>${env.GROUP_DEPLOY_TOKEN_NAME}</userName>
                    <password>${env.DEPLOY_TOKEN_VALUE}</password>
                </authenticationInfo>
            </configuration>
        </server>

and pom.xml like this;

    <repositories>
        <repository>
            <id>gitlab-saas-maven</id>
            <url>${env.CI_API_V4_URL}/groups/${env.TOP_GROUP_ID}/-/packages/maven</url>
        </repository>
    </repositories>

These settings worked fine for us for a long time until last week when they stopped. All the builds failed with errors saying package not found. After a lot of debugging in internal maven and tracing the http requests made by maven we discover that gitlab api is returning inconsistent (and incorrect in my opinion) response to maven request.

To explain the issue in details; when we use basic http authentication maven (internal httpclient) will make an unauthenticated call to the repository endpoint i.e. https://gitlab.com/api/v4/groups/<TOP_GROUP_ID>/-/packages/maven/com/dummy/git-scratchpad/2.8.0/git-scratchpad-2.8.0.pom. If that call returns 401 maven/httpclient inspects the response headers to identify which auth scheme is supported by the server based on WWW-Authenticate header; if the value is present starting with Basic * only then maven will send a second request but this time with the actual basic http auth as configured in the settings.xml.

For whatever reason gitlab has stopped returning 401 to the above url for this specific group when calling it unauthenticated. Gitlab is returning 404 package not found instead. Maven is taking it literally to mean the package is not there on this registry and fails the build. The kicker here is gitlab's behaviour is inconsistent, see below curl output;

❯ curl --head -s -o /dev/null -w "status: %{http_code}\nwww-authenticate: %header{www-authenticate}\n" https://gitlab.com/api/v4/groups/$TOP_GROUP_ID/-/packages/maven/com/dummy/git-scratchpad/2.8.0/git-scratchpad-2.8.0.pom
status: 404
www-authenticate:

❯ curl --head -s -o /dev/null -w "status: %{http_code}\nwww-authenticate: %header{www-authenticate}\n" https://gitlab.com/api/v4/groups/$SUB_GROUP_ID/-/packages/maven/com/dummy/git-scratchpad/2.8.0/git-scratchpad-2.8.0.pom
status: 401
www-authenticate: Basic realm="GitLab Packages Registry"

The subgroup is created within the top group and the deploy tokens are also created at top level. To evidence that the package actually exists in both registries see below output of curl commands which uses the base64 auth scheme;

❯ curl --head -s -o /dev/null -w "status: %{http_code}\ncontent-length: %header{content-length}\n" --user $GROUP_DEPLOY_TOKEN_NAME:$GROUP_DEPLOY_TOKEN_VALUE https://gitlab.com/api/v4/groups/$TOP_GROUP_ID/-/packages/maven/com/dummy/git-scratchpad/2.8.0/git-scratchpad-2.8.0.pom
status: 200
content-length: 3115

❯ curl --head -s -o /dev/null -w "status: %{http_code}\ncontent-length: %header{content-length}\n" --user $GROUP_DEPLOY_TOKEN_NAME:$GROUP_DEPLOY_TOKEN_VALUE https://gitlab.com/api/v4/groups/$SUB_GROUP_ID/-/packages/maven/com/dummy/git-scratchpad/2.8.0/git-scratchpad-2.8.0.pom
status: 200
content-length: 3115

We have tried the same settings with various version of maven i.e. 3.6.3, 3.8.8 and 3.9.9 with no different results. As far as I can tell by debugging maven source code and reviewing the network trace logs this is the way maven has always behaved i.e. sending the initial unauthenticated request to identify the auth schemes. And since the same settings worked for us previously we are suspecting something has changed on gitlab api or this group in particular for it to start returning 404 instead of 401. There are no settings at the group level that we can identify which could cause this.

Edited by 🤖 GitLab Bot 🤖