BuildKit fails to authenticate with dependency proxy due to missing account parameter in JWT request

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

BuildKit (using containerd) fails to authenticate with the GitLab dependency proxy, receiving a 403 Forbidden error when requesting JWT tokens. The issue occurs because BuildKit's authentication client does not include the account parameter in JWT authentication requests, which appears to be required by GitLab's dependency proxy.

Steps to reproduce

  1. Configure a GitLab CI/CD pipeline using BuildKit in rootless mode
  2. Attempt to pull images through the dependency proxy using CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX
  3. Configure authentication in ~/.docker/config.json as documented

Example .gitlab-ci.yml:

build_and_push:
  stage: build
  image:
    name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/moby/buildkit:rootless
    entrypoint: [""]
  variables:
    BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
  before_script:
    - |
      mkdir -p ~/.docker
      echo "{
        \"auths\": {
          \"${CI_REGISTRY}\": {
            \"auth\": \"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"
          },
          \"$(echo -n $CI_DEPENDENCY_PROXY_SERVER | awk -F[:] '{print $1}')\": {
            \"auth\": \"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"
          }
        }
      }" > ~/.docker/config.json
  script:
    - |
      buildctl-daemonless.sh build \
        --frontend dockerfile.v0 \
        --local context=. \
        --local dockerfile=. \
        --opt build-arg:REGISTRY=$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX \
        --output type=image,name=$CI_REGISTRY_IMAGE:latest,push=true

Dockerfile:

FROM ${REGISTRY}/archlinux/archlinux:base-devel

What is the current bug behavior?

BuildKit fails with the following error:

#4 [internal] load metadata for gitlab.example.com:443/group/dependency_proxy/containers/archlinux/archlinux:base-devel
#4 ERROR: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://gitlab.example.com/jwt/auth?scope=repository%3Agroup%2Fdependency_proxy%2Fcontainers%2Farchlinux%2Farchlinux%3Apull&service=dependency_proxy: 403 Forbidden

Access logs show the difference:

✅ Working (GitLab Runner pulling BuildKit image):

GET /jwt/auth?account=gitlab-ci-token&scope=repository%3Agroup%2Fdependency_proxy%2Fcontainers%2Fmoby%2Fbuildkit%3Apull&service=dependency_proxy HTTP/1.1" 200

❌ Failing (BuildKit requesting base image):

GET /jwt/auth?scope=repository%3Agroup%2Fdependency_proxy%2Fcontainers%2Farchlinux%2Farchlinux%3Apull&service=dependency_proxy HTTP/2.0" 403

The key difference: BuildKit's request is missing the account parameter.

What is the expected correct behavior?

One of the following should occur:

  1. GitLab's dependency proxy JWT endpoint should accept requests without the account parameter (following the Docker Registry v2 token authentication spec for anonymous/initial token requests), OR
  2. BuildKit should be documented as incompatible with the dependency proxy if the account parameter is mandatory

Relevant logs and/or screenshots

BuildKit error output:

error: failed to solve: failed to fetch anonymous token: unexpected status from GET request to https://gitlab.example.com/jwt/auth?scope=repository%3Agroup%2Fdependency_proxy%2Fcontainers%2Farchlinux%2Farchlinux%3Apull&service=dependency_proxy: 403 Forbidden

Environment

  • GitLab version: Self-managed (reported on 18.6)
  • GitLab Runner version: 18.6.3
  • BuildKit image: moby/buildkit:rootless
  • Configuration: Separate registry URL (registry_external_url different from main GitLab URL)

Additional Context

  • The same authentication credentials work perfectly with traditional Docker builds using docker buildx build
  • The GitLab Runner successfully pulls the BuildKit image itself from the dependency proxy
  • The container registry authentication works fine (BuildKit can push to CI_REGISTRY)
  • Only BuildKit's requests to pull images through the dependency proxy fail

This suggests the issue is specific to how BuildKit/containerd constructs JWT authentication requests versus how Docker CLI does it, and how GitLab's dependency proxy validates these requests.

Possible fixes

  1. Update GitLab's dependency proxy JWT authentication to handle requests without the account parameter
  2. Investigate if this is a regression or if the account parameter was always required
  3. Update documentation to clarify BuildKit compatibility with dependency proxy

Related Documentation

Edited by 🤖 GitLab Bot 🤖