Pulling from gitlab registry : unable to decode token response: invalid character '<' looking for beginning of value

Issue Summary

Docker pull operations are failing with the error: unable to decode token response: invalid character '<' looking for beginning of value

This occurs when containerd v2.2.0+ (Docker 29.1.x) attempts to access the OCI Referrers API (*/referrers/* paths) on the container registry. GitLab is not treating these paths as container registry endpoints and is redirecting containerd to /users/sign_in, returning an HTML sign-in page instead of a proper 404 response.

Root Cause

When containerd attempts to GET the referrers file using its JWT token, GitLab redirects the request to /users/sign_in and returns a 200 response with HTML content. Containerd's oauth2 library tries to parse this HTML as JSON before checking the HTTP status code, resulting in the "invalid character '<'" error.

Implementation Plan

Add a catch-all route for */referrers/* API paths in the container registry routing to return a proper 404 response instead of redirecting to the sign-in page.

Scope: Limit the catch-all route to specific OCI spec paths (e.g., */referrers/*) to:

  • Prevent unintended side effects from overly broad matching
  • Still catch typos or malformed requests appropriately
  • Maintain compatibility with future OCI spec additions

Location: config/routes/group.rb - Add a fallback route to the dependency proxy handlers

Expected Behavior: When containerd requests */referrers/* paths with a valid JWT token, GitLab should return a proper 404 JSON response instead of redirecting to the sign-in page.

Related Information

  • Root cause identified: containerd v2.2.0+ (Docker 29.1.x) includes OCI Referrers support
  • Upstream PR: https://github.com/containerd/containerd/pull/12857
  • Workaround: Pin Docker to version 29.0.4 (temporary solution)
  • Customer Impact: Multiple customers affected, pinning to older Docker versions is not a viable long-term solution

Original Issue Description:

Hello,

I'm encountering an issue while trying to pull a docker image from our private Gitlab Registry :

docker pull gitlab.tld:4567/fusioniam/fusioniam/fusioniam-lemonldap-ng:master-2f6be426
Error response from daemon: Head "https://gitlab.tld:4567/v2/fusioniam/fusioniam/fusioniam-lemonldap-ng/manifests/master-2f6be426": unable to decode token response: invalid character '<' looking for beginning of value

The project is public, there is no restriction whatsoever. I should have been able to pull this image without docker login, but I was able to successfuly log in:

$ docker login gitlab.tld
Username: user
Password: 

WARNING! Your credentials are stored unencrypted in '/home/user/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/

Login Succeeded

Trying to log to gitlab.tld:4567 does not work though:

$ docker login gitlab.tld:4567
Username: user
Password: 
Error response from daemon: Get "https://gitlab.tld:4567/v2/": unable to decode token response: invalid character '<' looking for beginning of value

Running Gitlab-CE Omnibus package 18.6.0-ce.0 on Debian 12 64bits.

Edited by 🤖 GitLab Bot 🤖