DOCKER_AUTH_CONFIG is now also processed by Docker CLI

TL;DR

Starting from Docker CLI v28.3.0, it also uses DOCKER_AUTH_CONFIG environment variable, which may result in unexpected conflicts in jobs.

Summary

A few days ago I upgraded GitLab runner on a few of my hosts to version 18.1.1. Those runners are used as shell runners. After the upgrade some of my jobs, which have docker push commands, unexpectedly started to fail with the following message:

unauthorized: access to the requested resource is not authorized

My projects have DOCKER_AUTH_CONFIG variable set with read-only credentials to pull docker images from a self-hosted Nexus docker registry. Some jobs, which run docker push commands use read-write credentials supplied via a configuration file, e.g. ~/.docker/config.json. Suddenly after the upgrade this setup stopped working with the above error message.

I started to diagnose the issue and quickly found out that it relates to the DOCKER_AUTH_CONFIG variable. I looked for changes in GitLab runner source code diffs between my previous used version and the 18.1.1, but found no changed related to the DOCKER_AUTH_CONFIG variable.

Then I remembered, that together with the runner upgrade I've also upgraded system packages on the underlying hosts. And one of the packages that were upgraded was docker-ce-cli (from version 28.2.2 to 28.3.0).

So I've looked into the Docker CLI diffs and discovered that the new version of the CLI also favors the DOCKER_AUTH_CONFIG variable. You can find the diffs here.

As I now know the reason for the failure, I was able to fix it in my pipeline jobs. What concerns me though is that this variable conflict may now be a source for many different unexpected errors for many users who use Docker CLI in their jobs with a similar setups, especially taking into consideration GitLab's specific format for the variable contents, I'm not sure Docker CLI is able to use it the same way as GitLab runner does.

I believe this at least should be mentioned now in GitLab docs. Thanks.