Skip to content

Dependency proxy is not aware of relative URL

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

On self-managed instance under relative URL, docker login returns 404 Not Found.
Although a custom nginx setting allows login succeed, docker pull still returns HTTP 404.
After reverting external_url back to non-relative URL, dependency proxy works as expected.

Workaround

Add a custom nginx configuration to their gitlab.rb:

nginx['custom_gitlab_server_config'] = "location /v2 {\n    proxy_cache off;\n    proxy_http_version 1.1;\n    proxy_pass  http://gitlab-workhorse/gitlab/v2;\n  }\n  location /v2/gitlab {\n    proxy_cache off;\n    proxy_http_version 1.1;\n    proxy_pass  http://gitlab-workhorse/gitlab/v2;\n  }\n"

Then run gitlab-ctl reconfigure.

Steps to reproduce

  1. Configure relative URL
  2. docker login <GitLab app relative URL> -u <username> -p <password>
    -> fails with 404
  3. Configure custom nginx setting.
  4. docker login <GitLab app relative URL> -u <username> -p <password>
    -> succeeds
  5. docker pull <image prefix shown on the group's dependency proxy page>/<image name>:<tag>
    -> fails with 404

What is the current bug behavior?

(Credential warnings from docker are omitted below)

With external_url 'https://localhost/gitlab':

$ docker login -u root -p '********' https://localhost
Error response from daemon: login attempt to http://localhost/v2/ failed with status: 404 Not Found

With below custom nginx setting:

nginx['custom_gitlab_server_config'] = "location /v2 {\n    proxy_cache off;\n    proxy_http_version 1.1;\n    proxy_pass  http://gitlab-workhorse/gitlab/v2;\n  }\n"
$ docker login -u root -p '********' https://localhost

Login Succeeded
$ docker pull localhost/gitlab/dep-proxy-test/dependency_proxy/containers/ubuntu:latest
Error response from daemon: error parsing HTTP 404 response body: unexpected end of JSON input: ""

Dependency proxy screen:
image

What is the expected correct behavior?

docker login succeeds under relative URL without custom nginx setting.
docker pull succeeds under relative URL.

Relevant logs and/or screenshots

`gitlab-ctl tail` output excerpt during failing `docker pull` (gitlab-rails, gitlab-workhorse and nginx only) 
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/gitlab/v2","format":"html","controller":"Groups::DependencyProxyAuthController","action":"authenticate","status":401,"time":"2023-08-11T11:34:06.999Z","params":[],"remote_ip":"172.20.0.1","ua":"docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \\(linux\\))","queue_duration_s":0.129619,"request_urgency":"low","target_duration_s":5,"redis_calls":5,"redis_duration_s":0.008948,"redis_read_bytes":1014,"redis_write_bytes":389,"redis_cache_calls":5,"redis_cache_duration_s":0.008948,"redis_cache_read_bytes":1014,"redis_cache_write_bytes":389,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.013,"db_main_duration_s":0.013,"db_main_replica_duration_s":0.0,"cpu_s":0.142433,"mem_objects":7634,"mem_bytes":1213311,"mem_mallocs":3600,"mem_total_bytes":1518671,"pid":782,"worker_id":"puma_7","rate_limiting_gates":[],"correlation_id":"01H7J602QRBC675PWVDQY48K2R","db_duration_s":0.0,"view_duration_s":0.0037,"duration_s":0.01722}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"text/plain; charset=utf-8","correlation_id":"01H7J602QRBC675PWVDQY48K2R","duration_ms":191,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":401,"system":"http","time":"2023-08-11T11:34:07Z","ttfb_ms":191,"uri":"/gitlab/v2/","user_agent":"docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \\(linux\\))","written_bytes":0}

==> /var/log/gitlab/nginx/gitlab_access.log <==
172.20.0.1 - - [11/Aug/2023:11:34:07 +0000] "GET /v2/ HTTP/1.1" 401 0 "" "docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \x5C(linux\x5C))" -

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/gitlab/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":200,"time":"2023-08-11T11:34:07.667Z","params":[{"key":"account","value":"root"},{"key":"scope","value":"repository:gitlab/tests/dependency_proxy/containers/centos:pull"},{"key":"service","value":"dependency_proxy"}],"correlation_id":"01H7J602ZCG0DPBYSRDCJQKR33","meta.caller_id":"JwtController#auth","meta.remote_ip":"172.20.0.1","meta.feature_category":"system_access","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","remote_ip":"172.20.0.1","user_id":1,"username":"root","ua":"docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \\(linux\\))","queue_duration_s":0.179187,"request_urgency":"low","target_duration_s":5,"redis_calls":11,"redis_duration_s":0.016381,"redis_read_bytes":2149,"redis_write_bytes":718,"redis_cache_calls":11,"redis_cache_duration_s":0.016381,"redis_cache_read_bytes":2149,"redis_cache_write_bytes":718,"db_count":9,"db_write_count":0,"db_cached_count":1,"db_replica_count":0,"db_primary_count":9,"db_main_count":9,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_main_cached_count":1,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.07,"db_main_duration_s":0.07,"db_main_replica_duration_s":0.0,"cpu_s":0.519685,"mem_objects":24161,"mem_bytes":3579396,"mem_mallocs":13257,"mem_total_bytes":4545836,"pid":780,"worker_id":"puma_6","rate_limiting_gates":[],"db_duration_s":0.05288,"view_duration_s":0.00059,"duration_s":0.39269}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"application/json; charset=utf-8","correlation_id":"01H7J602ZCG0DPBYSRDCJQKR33","duration_ms":590,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2023-08-11T11:34:07Z","ttfb_ms":590,"uri":"/gitlab/jwt/auth?account=root\u0026scope=repository%3Agitlab%2Ftests%2Fdependency_proxy%2Fcontainers%2Fcentos%3Apull\u0026service=dependency_proxy","user_agent":"docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \\(linux\\))","written_bytes":239}

==> /var/log/gitlab/nginx/gitlab_access.log <==
172.20.0.1 - root [11/Aug/2023:11:34:07 +0000] "GET /gitlab/jwt/auth?account=root&scope=repository%3Agitlab%2Ftests%2Fdependency_proxy%2Fcontainers%2Fcentos%3Apull&service=dependency_proxy HTTP/1.1" 200 239 "" "docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \x5C(linux\x5C))" -

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"HEAD","path":"/gitlab/v2/gitlab/tests/dependency_proxy/containers/centos/manifests/latest","format":"json","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":404,"time":"2023-08-11T11:34:07.966Z","params":[{"key":"group_id","value":"gitlab/tests"},{"key":"image","value":"centos"},{"key":"tag","value":"latest"}],"correlation_id":"01H7J603J4MJSD791ZCZFZBQT0","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"172.20.0.1","meta.feature_category":"dependency_proxy","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","remote_ip":"172.20.0.1","user_id":1,"username":"root","ua":"docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \\(linux\\))","queue_duration_s":0.027198,"request_urgency":"low","target_duration_s":5,"redis_calls":2,"redis_duration_s":0.006277,"redis_read_bytes":203,"redis_write_bytes":354,"redis_cache_calls":1,"redis_cache_duration_s":0.000977,"redis_cache_read_bytes":203,"redis_cache_write_bytes":58,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.0053,"redis_shared_state_write_bytes":296,"db_count":4,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":4,"db_main_count":4,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.066,"db_main_duration_s":0.066,"db_main_replica_duration_s":0.0,"cpu_s":0.22314,"mem_objects":8430,"mem_bytes":663056,"mem_mallocs":2999,"mem_total_bytes":1000256,"pid":776,"worker_id":"puma_4","rate_limiting_gates":[],"db_duration_s":0.06447,"view_duration_s":0.0,"duration_s":0.24154}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"application/json","correlation_id":"01H7J603J4MJSD791ZCZFZBQT0","duration_ms":297,"host":"localhost","level":"info","method":"HEAD","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":404,"system":"http","time":"2023-08-11T11:34:07Z","ttfb_ms":297,"uri":"/gitlab/v2/gitlab/tests/dependency_proxy/containers/centos/manifests/latest","user_agent":"docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \\(linux\\))","written_bytes":0}

==> /var/log/gitlab/nginx/gitlab_access.log <==
172.20.0.1 - - [11/Aug/2023:11:34:07 +0000] "HEAD /v2/gitlab/tests/dependency_proxy/containers/centos/manifests/latest HTTP/1.1" 404 0 "" "docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \x5C(linux\x5C))" -

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/gitlab/v2/gitlab/tests/dependency_proxy/containers/centos/manifests/latest","format":"json","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":404,"time":"2023-08-11T11:34:08.141Z","params":[{"key":"group_id","value":"gitlab/tests"},{"key":"image","value":"centos"},{"key":"tag","value":"latest"}],"correlation_id":"01H7J603VT5W7R03SZ08TQNH4E","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"172.20.0.1","meta.feature_category":"dependency_proxy","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","remote_ip":"172.20.0.1","user_id":1,"username":"root","ua":"docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \\(linux\\))","queue_duration_s":0.075069,"request_urgency":"low","target_duration_s":5,"redis_calls":8,"redis_duration_s":0.008101,"redis_read_bytes":1335,"redis_write_bytes":794,"redis_cache_calls":7,"redis_cache_duration_s":0.006739,"redis_cache_read_bytes":1335,"redis_cache_write_bytes":498,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.001362,"redis_shared_state_write_bytes":296,"db_count":6,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":6,"db_main_count":6,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.022,"db_main_duration_s":0.022,"db_main_replica_duration_s":0.0,"cpu_s":0.120714,"mem_objects":13457,"mem_bytes":1681180,"mem_mallocs":5627,"mem_total_bytes":2219460,"pid":772,"worker_id":"puma_2","rate_limiting_gates":[],"db_duration_s":0.01123,"view_duration_s":0.0,"duration_s":0.06668}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"application/json","correlation_id":"01H7J603VT5W7R03SZ08TQNH4E","duration_ms":154,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":404,"system":"http","time":"2023-08-11T11:34:08Z","ttfb_ms":153,"uri":"/gitlab/v2/gitlab/tests/dependency_proxy/containers/centos/manifests/latest","user_agent":"docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \\(linux\\))","written_bytes":0}

==> /var/log/gitlab/nginx/gitlab_access.log <==
172.20.0.1 - - [11/Aug/2023:11:34:08 +0000] "GET /v2/gitlab/tests/dependency_proxy/containers/centos/manifests/latest HTTP/1.1" 404 0 "" "docker/20.10.21 go/go1.18.1 git-commit/20.10.21-0ubuntu1~20.04.2 kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.21 \x5C(linux\x5C))"

Output of checks

Results of GitLab environment info

  • Customer environment

    • RHEL 7.9
    • GitLab 15.11.3-ee / Linux package

  • My reproducing environment

    • Ubuntu 20.04 on WSL2 / Windows 10
    • Docker 20.10
    • 15.11.3-ee and 16.2.3-ee / Official Docker images
Expand for output related to GitLab environment info 
System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.0.6p216
Gem Version:    3.4.14
Bundler Version:2.4.16
Rake Version:   13.0.6
Redis Version:  7.0.12
Sidekiq Version:6.5.7
Go Version:     unknown

GitLab information
Version:        16.2.3-ee
Revision:       7ac2b0a343c
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.11
URL:            https://localhost/gitlab
HTTP Clone URL: https://localhost/gitlab/some-group/some-project.git
SSH Clone URL:  ssh://git@localhost:10022/some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        14.23.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 14.23.0 ? ... OK (14.23.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
4/2 ... yes
4/3 ... yes
8/4 ... yes
8/7 ... yes
7/9 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Edited by Jaime Martinez