Dependency proxy in pipeline fails with error HTTP 599
Summary
CI Pipeline job fails when configured to use the dependency proxy to fetch docker images, a HTTP 599 error is returned.
If I remove the ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX} from the job it works so proxy is working at least in this configuration.
Thanks in advance for the support.
Steps to reproduce
Configure a CI job to build a python package.
Example Project
I configured a simple job to build and publish in gitlab package repository a python package:
variables:
HTTP_PROXY: http://proxy.domain:8080
HTTPS_PROXY: http://proxy.domain:8080
NO_PROXY: ... own domain ...
stages:
- package
package:
stage: package
image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/python:3.11-slim-buster
script:
- pip install build twine
- python -m build
- TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
only:
- tagsAs my gitlab instance is running behind the organisation proxy, I set the config in /etc/gitlab/gitlab.rb:
# gitlab_workhorse['env_directory'] = "/opt/gitlab/etc/gitlab-workhorse/env"
gitlab_workhorse['env'] = {
# 'PATH' => "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/bin:/usr/bin",
# 'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/"
"http_proxy" => "http://proxy.domain:8080",
"https_proxy" => "http://proxy.domain:8080"
}What is the current bug behavior?
Error response from daemon: received unexpected HTTP status: 599 status code 599 (manager.go:237:20s)
What is the expected correct behavior?
No error, job goes through.
Relevant logs and/or screenshots
Here is the job result:
Running with gitlab-runner 16.2.0 (782e15da)
2 on ana-app-gitlab docker shared FyDaqi1-, system ID: s_1108ee710418
3 Preparing the "docker" executor
4Using Docker executor with image git.domain:443/it/dependency_proxy/containers/python:3.11-slim-buster ...
5Authenticating with credentials from job payload (GitLab Registry)
6Pulling docker image git.domain:443/it/dependency_proxy/containers/python:3.11-slim-buster ...
7WARNING: Failed to pull image with policy "always": Error response from daemon: received unexpected HTTP status: 599 status code 599 (manager.go:237:20s)
8ERROR: Job failed: failed to pull image "git.domain:443/it/dependency_proxy/containers/python:3.11-slim-buster" with specified policies [always]: Error response from daemon: received unexpected HTTP status: 599 status code 599 (manager.go:237:20s)
Output of checks
Results of GitLab environment info
System information
System: Ubuntu 22.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 3.0.6p216
Gem Version: 3.4.14
Bundler Version:2.4.16
Rake Version: 13.0.6
Redis Version: 7.0.12
Sidekiq Version:6.5.7
Go Version: unknown
GitLab information
Version: 16.2.3-ee
Revision: 7ac2b0a343c
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 13.11
URL: https://git.domain
HTTP Clone URL: https://git.domain/some-group/some-project.git
SSH Clone URL: git@git.domain:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.23.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shellExpand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.23.0 ? ... OK (14.23.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
4/1 ... yes
4/2 ... yes
7/3 ... yes
8/4 ... yes
3/5 ... yes
3/6 ... yes
11/7 ... yes
4/8 ... yes
4/9 ... yes
3/10 ... yes
3/11 ... yes
3/12 ... yes
3/13 ... yes
3/14 ... yes
3/15 ... yes
3/17 ... yes
3/18 ... yes
3/19 ... yes
4/20 ... yes
7/21 ... yes
18/22 ... yes
18/23 ... yes
3/24 ... yes
3/25 ... yes
3/26 ... yes
3/27 ... yes
3/28 ... yes
3/29 ... yes
3/30 ... yes
3/31 ... yes
3/32 ... yes
17/33 ... yes
3/34 ... yes
3/35 ... yes
4/36 ... yes
4/38 ... yes
3/39 ... yes
3/40 ... yes
3/41 ... yes
3/42 ... yes
17/43 ... yes
3/44 ... yes
3/45 ... yes
37/46 ... yes
3/47 ... yes
3/48 ... yes
39/49 ... yes
39/50 ... yes
3/51 ... yes
3/52 ... yes
3/53 ... yes
18/54 ... yes
3/55 ... yes
3/56 ... yes
3/57 ... yes
18/58 ... yes
51/59 ... yes
51/60 ... yes
18/61 ... yes
61/62 ... yes
61/63 ... yes
127/64 ... yes
25/65 ... yes
3/66 ... yes
3/68 ... yes
33/69 ... yes
33/70 ... yes
33/71 ... yes
3/72 ... yes
3/73 ... yes
143/74 ... yes
145/75 ... yes
3/76 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 31
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished