Gitlab Container Registry will not honor http_proxy when accessing S3 backend

Running Gitlab via Helm charts in a Kubernetes environment - so the container registry is running in containers, currently based on image registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v4.5.0-gitlab

Switched to using a private S3 instance for storage. Both the consolidated storage configuration and the registry-specific configuration was created and is effective.

S3 Storage access seems to work, except for the container registry, where requests time out. Debugging confirmed that the requests sent by the container registry are sent to the configured S3 host directly, not honoring the configured http_proxy etc variables. The S3 end point is only accessible via the configured proxy, hence the time-out.

Configuring S3 debug within the registry storage configuration shows the requests being correctly created at the "URL" level:

"config_http_addr":":5000","config_http_host":"","config_http_net":"","config_http_prefix":"","config_http_relative_urls":false,"correlation_id":"01J5KN1Z1A2A87PFCWTKPDZYRB","go_version":"go1.22.4","level":"info","method":"GET","msg":"router info","path":"/v2/customers/***redacted***/tags/list","root_repo":"customers","router":"gorilla/mux","time":"2024-08-18T21:07:26.378Z","vars_name":"customers/***redacted***","version":"v4.5.0-gitlab"}
{"auth_project_paths":["customers/***redacted***"],"auth_user_name":"","auth_user_type":"","correlation_id":"01J5KN1Z1A2A87PFCWTKPDZYRB","go_version":"go1.22.4","level":"info","msg":"authorized request","root_repo":"customers","time":"2024-08-18T21:07:26.381Z","vars_name":"customers/***redacted***","version":"v4.5.0-gitlab"}
2024/08/18 21:07:26 DEBUG: Request s3/ListObjectsV2 Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /dmzgitlab-registry?delimiter=%2F&list-type=2&max-keys=1000&prefix=docker%2Fregistry%2Fv2%2Frepositories%2Fcustomers%2F***redacted***%2F_manifests%2Ftags%2F HTTP/1.1
Host: ndergw.***redacted***:7480
User-Agent: docker-distribution/v4.5.0-gitlab (go1.22.4) aws-sdk-go/1.46.7 (go1.22.4; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=***redacted***/20240818/default/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=***redacted***
X-Amz-Content-Sha256: ***redacted***
X-Amz-Date: 20240818T210726Z
Accept-Encoding: gzip


-----------------------------------------------------
{"delay_s":0.439227405,"error":"RequestCanceled: request context canceled\ncaused by: context canceled","level":"info","msg":"S3: retrying after error","time":"2024-08-18T21:07:46.391Z"}
{"auth_project_paths":["customers/***redacted***"],"auth_user_name":"","auth_user_type":"","code":"REQUESTCANCELED","correlation_id":"01J5KN1Z1A2A87PFCWTKPDZYRB","detail":"context canceled","error":"requestcanceled: request canceled","go_version":"go1.22.4","host":"gitlab-registry.gitlab.svc:5000","level":"error","method":"GET","msg":"request canceled","remote_addr":"10.42.2.136:37618","root_repo":"customers","time":"2024-08-18T21:07:46.831Z","uri":"/v2/customers/***redacted***/tags/list?n=10000","user_agent":"GitLab/17.1.2","vars_name":"customers/***redacted***","version":"v4.5.0-gitlab"}
{"content_type":"application/json","correlation_id":"01J5KN1Z1A2A87PFCWTKPDZYRB","duration_ms":20452,"host":"gitlab-registry.gitlab.svc:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.42.2.136:37618","remote_ip":"10.42.2.136","status":400,"system":"http","time":"2024-08-18T21:07:46.831Z","ttfb_ms":20452,"uri":"/v2/customers/***redacted***/tags/list?n=10000","user_agent":"GitLab/17.1.2","written_bytes":81}

tcpdump'ing on the host where the registry container is run, the direct TCP requests to ndergw.redacted port 7480 can be seen (without answer, as the traffic is blocked by infrastructure firewalling). All S3 requests by other gitlab components/containers go via the HTTP proxy configured in all containers, and can be seen in the proxy's access log.

The gitlab container registry container has the according environment variables, curl used from within the container will honor these and can access the S3 end point. The gitlab container registry container's /etc/docker/registry/config.yml was extended to also have an "env:" section, to no avail:

env:
  http_proxy: http://192.168.163.17:3129
  https_proxy: http://192.168.163.17:3129
  no_proxy: localhost,127.0.0.1,10.0.0.0/8,172.16.0.0/22,192.168.163.0/24,.svc,.cluster.local,169.254.169.254

The S3 storage configuration is present in that same file /etc/docker/registry/config.yml:

storage:
  # configuration of registry `storage` secret for RadosGW
  #   See https://docs.docker.com/registry/storage-drivers/s3
  #   See https://gitlab.com/gitlab-org/charts/gitlab/tree/master/doc/charts/registry/#storage
  #   See https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-object-storage
  #   See https://gitlab.com/gitlab-org/container-registry/-/tree/master/docs-gitlab#additional-parameters
  s3:
    region: default
    accesskey: ***redacted***
    secretkey: ***redacted***
    regionendpoint: "https://ndergw.***redacted***:7480"
    skipverify: true
    v4auth: true
    forcepathstyle: true
    bucket: dmzgitlab-registry
    loglevel: LogDebug
  
  redirect:
    disable: true
  
  delete:
    enabled: true
  maintenance:
    readonly:
      enabled: false
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false

The proxy-defining variables are present both in upper-case and lower-case variants in the container's environment, to rule out uses of only one of the two variants:

git@gitlab-registry-649c6c98b9-j28bq:/$ env|grep -i proxy|sort
HTTPS_PROXY=http://192.168.163.17:3129
HTTP_PROXY=http://192.168.163.17:3129
NO_PROXY=localhost,127.0.0.1,10.0.0.0/8,172.16.0.0/22,192.168.163.0/24,.svc,.cluster.local,169.254.169.254
http_proxy=http://192.168.163.17:3129
https_proxy=http://192.168.163.17:3129
git@gitlab-registry-649c6c98b9-j28bq:/$

(Having no no_proxy (lower-case) should not keep the code from using the proxy settings, as no exceptions are defined.)

This kind of error (http_proxy not honored in gitlab registry) has popped up in other issue reports, too:

I could not see references to an actual fix.

As I have a reproducer up&running at the moment, I can offer to provide further debug information and run tests.

Edited Aug 18, 2024 by J Mo