Dependency proxy on self-hosted instance returning "500 Internal Error"

Summary

I have a self-hosted Gitlab Ultimate trial instance deployed behind a company proxy. I need to make the dependency proxy fetch images from docker hub. I've configured http_proxy variables where necessary but am not able to get it running in my ci pipeline.

The logs say something is timing out but with the proxy set I see no reason for that.

What I've tried:

  • emptying dependency_proxy_* talbes in postgres as per #354574 (closed)
  • setting http and https proxy in environment variables
  • restarting gitlab container

Steps to reproduce

  1. Run a self-hosted Gitlab instance in Docker Compose
  2. Create a group and within it an (empty) project
  3. Register a runner with a Docker executor
  4. Create a CI pipeline
  5. Within the pipeline, pull an image from the dependency proxy

What is the current bug behavior?

The pipeline fails with "500: Internal Server Error":

$ docker pull $CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/alpine:latest
Error response from daemon: received unexpected HTTP status: 500 Internal Server Error

What is the expected correct behavior?

The Dependency proxy should fetch the requested image from Docker hub.

Relevant logs and/or screenshots

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/v2/pavlis/dependency_proxy/containers/python/manifests/slim-bookworm","format":"json","controller":"Groups::DependencyProxyForContainersCo
ntroller","action":"manifest","status":500,"time":"2024-06-19T06:14:04.066Z","params":[{"key":"group_id","value":"pavlis"},{"key":"image","value":"python"},{"key":
"tag","value":"slim-bookworm"}],"correlation_id":"01J0QJ0S6Y4D7YPYJ2RPP6JP6R","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remo
te_ip":"<redacted>","meta.feature_category":"dependency_proxy","meta.user":"root","meta.user_id":1,"meta.root_namespace":"pavlis","meta.client_id":"user/1","remo
te_ip":"<redacted>","user_id":1,"username":"root","ua":"docker/26.1.2 go/go1.21.10 git-commit/ef1912d kernel/6.1.0-21-amd64 os/linux arch/amd64 UpstreamClient(Do
cker-Client/26.1.4 \\(linux\\))","queue_duration_s":0.041215,"request_urgency":"low","target_duration_s":5,"redis_calls":12,"redis_duration_s":0.005293,"redis_read
_bytes":2102,"redis_write_bytes":1090,"redis_cache_calls":1,"redis_cache_duration_s":0.001033,"redis_cache_read_bytes":113,"redis_cache_write_bytes":96,"redis_clus
ter_shared_state_calls":1,"redis_cluster_shared_state_duration_s":0.000613,"redis_cluster_shared_state_write_bytes":296,"redis_feature_flag_calls":10,"redis_featur
e_flag_duration_s":0.003647,"redis_feature_flag_read_bytes":1989,"redis_feature_flag_write_bytes":698,"db_count":11,"db_write_count":0,"db_cached_count":0,"db_repl
ica_count":0,"db_primary_count":11,"db_main_count":11,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cac
hed_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_w
al_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cac
hed_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s
":0.0,"db_primary_duration_s":0.013,"db_main_duration_s":0.013,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"external_htt
p_count":1,"external_http_duration_s":0.1264873370528221,"cpu_s":5.627535,"mem_objects":24070,"mem_bytes":3063316,"mem_mallocs":9196,"mem_total_bytes":4026116,"pid
":729,"worker_id":"puma_3","rate_limiting_gates":[],"exception.class":"Rack::Timeout::RequestTimeoutException","exception.message":"Request ran for longer than 600
00ms ","exception.backtrace":["lib/gitlab/http.rb:46:in `public_send'","lib/gitlab/http.rb:46:in `block (2 levels) in singleton class'","app/services/dependency_pr
oxy/head_manifest_service.rb:14:in `execute'","app/services/dependency_proxy/find_cached_manifest_service.rb:19:in `execute'","app/controllers/groups/dependency_pr
oxy_for_containers_controller.rb:23:in `manifest'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/app/controllers/ee/application_controller.rb:45:in `set_cur
rent_ip_address'","app/controllers/application_controller.rb:498:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_
controller.rb:489:in `set_session_storage'","lib/gitlab/i18n.rb:114:in `with_locale'","lib/gitlab/i18n.rb:120:in `with_user_locale'","app/controllers/application_c
ontroller.rb:480:in `set_locale'","app/controllers/application_controller.rb:473:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:
in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middle
ware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_mi
ddleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/middleware/
go.rb:20:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:37:in `within'","lib/gitlab/middleware/q
uery_analyzer.rb:11:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middlewa
re/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/path_traversal_check.rb:48:in `call'","lib/gitlab/mi
ddleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.r
b:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","config/initializers/fi
x_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `
call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","lib/gitlab/middleware/release_env.
rb:13:in `call'"],"db_duration_s":0.00918,"view_duration_s":0.0,"duration_s":73.49571}

GitLab environment info

Gitlab image: gitlab-ee:16.6.1-ee.0

docker-compose.yaml

services:
  web:
    image: 'gitlab/gitlab-ee:16.6.1-ee.0'
    restart: always
    hostname: '<gitlab-ip-redacted>'
    environment:
      http_proxy: "http://<proxy-redacted>:8888"
      https_proxy: "http://<proxy-redacted>:8888"
      no_proxy: "<gitlab-ip-redacted>,localhost,127.0.0.1,1,2,3,4,5,6,7,8,9,0"
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://<gitlab-ip-redacted>'
        logrotate['enable'] = true
        logrotate['log_directory'] = "/var/log/gitlab/logrotate"
        logging['svlogd_num'] = 2 # keep 30 rotated log files
        logging['logrotate_frequency'] = "daily"
        logging['logrotate_maxsize'] = 200 * 1024 * 1024 
        logging['logrotate_rotate'] = 2 

        registry_external_url 'https://<gitlab-ip-redacted>:5050'
        gitlab_rails['dependency_proxy_enabled'] = true

        gitlab_kas_external_url "ws://<gitlab-ip-redacted>/-/kubernetes-agent/"
        gitlab_kas['enable'] = true

        gitlab_rails['env'] = {
            "http_proxy" => "http://<proxy-redacted>:8888",
            "https_proxy" => "http://<proxy-redacted>:8888",
            "no_proxy" => "<gitlab-ip-redacted>,localhost,127.0.0.1,1,2,3,4,5,6,7,8,9,0"
        }
        gitlab_workhorse['env'] = {
            "http_proxy" => "http://<proxy-redacted>:8888",
            "https_proxy" => "http://<proxy-redacted>:8888",
            "no_proxy" => "<gitlab-ip-redacted>,localhost,127.0.0.1,1,2,3,4,5,6,7,8,9,0"
        }

        # If you use the docker registry
        registry['env'] = {
            "http_proxy" => "http://<proxy-redacted>:8888",
            "https_proxy" => "http://<proxy-redacted>:8888",
            "no_proxy" => "<gitlab-ip-redacted>,localhost,127.0.0.1,1,2,3,4,5,6,7,8,9,0"
        }
    ports:
      - '80:80'   # web ui
      - '443:443' # web ui
      - '22:22'   # ssh
      - '5050:5050' # registry
    volumes:
      # Umiestnenie GitLab dat
      - '/root/gitlab/gl-data3/config:/etc/gitlab'
      - '/root/gitlab/gl-data3/logs:/var/log/gitlab'
      - '/root/gitlab/gl-data3/data:/var/opt/gitlab'
    shm_size: '256m'

  runner1:
    image: 'gitlab/gitlab-runner:alpine3.18'
    restart: always
    hostname: 'runner1'
    volumes:
      - '/var/lib/docker/gitlab/gitlab-runner/config:/etc/gitlab-runner'
      - '/var/run/docker.sock:/var/run/docker.sock'
      - '/root/gitlab/certs/ca:/etc/gitlab-runner/certs'
      - '/root/gitlab/certs/ca/<gitlab-ip-redacted>.crt:/etc/gitlab-runner/custom-ca/<gitlab-ip-redacted>.crt'

Content of /etc/gitlan-runner/config/config.toml

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "dind-tls"
  url = "https://<gitlab-ip-redacted>"
  id = 6
  token = "<redacted?"
  token_obtained_at = 2024-06-17T14:47:24Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    Insecure = false
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:24.0.5"
    privileged = true
    pull_policy = "if-not-present"
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache" ]
    shm_size = 0
    network_mtu = 0

Content of .gitlab-ci.yml:

stages:
- build 

build:
  stage: build
  image:
    name: docker:26.1.4
  before_script:
  - echo $REGISTRY_CERT | base64 -d > /usr/local/share/ca-certificates/registry.crt
  - update-ca-certificates
  - docker login -u $CI_DEPENDENCY_PROXY_USER -p $CI_DEPENDENCY_PROXY_PASSWORD $CI_DEPENDENCY_PROXY_SERVER
  script:
  - docker pull $CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/alpine:latest

Results of GitLab application Check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.30.0 ? ... OK (14.30.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
1/1 ... yes
11/2 ... yes
11/3 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 3.0.6 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 5
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished
Edited by 🤖 GitLab Bot 🤖