Skip to content

Dependency Proxy not falling back on timeout

Summary

We are currently having issues with connecting to the docker hub. We're not sure why that happens but randomly we get a TSL handshake failed and also just regular timeouts. We're using the dependency proxy and thought that this would help us.

However the proxy fails aswell with the error:

error pulling image configuration: received unexpected HTTP status: 599 status code 599

when using docker pull.

According to the code it seems like the 599 comes from this function: https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/dependency_proxy/find_or_create_manifest_service.rb#L19

And according to the cached_manifest_matches function if there is an error then it won't use the cached manifest and the 599 is generated by gitlab itself on a timeout here: https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/dependency_proxy/head_manifest_service.rb#L22

In my opinion this won't trigger: https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/dependency_proxy/find_or_create_manifest_service.rb#L29

Also the controller currently fails with 599 if no token can be fetched: https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/groups/dependency_proxy_for_containers_controller.rb#L95

Steps to reproduce

  1. Pull image from dependency proxy
  2. Turn off access to registry-1.docker.io
  3. Pull again

What is the current bug behavior?

Unable to pull image from dependency proxy even so it is cached

What is the expected correct behavior?

On timeout to docker.io provide the cached manifest

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

System information
System:         Ubuntu 18.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.14
Git Version:    2.32.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        14.2.3-ee
Revision:       b5eea856eca
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://xxx.com
HTTP Clone URL: https://xxx.com/some-group/some-project.git
SSH Clone URL:  git@xxx.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.19.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.19.1 ? ... OK (13.19.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 45 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
6/3 ... yes
6/4 ... yes
6/5 ... yes
6/6 ... yes
12/7 ... yes
12/8 ... yes
14/9 ... yes
6/10 ... yes
6/11 ... yes
12/12 ... yes
6/18 ... yes
21/22 ... yes
17/23 ... yes
17/24 ... yes
17/26 ... yes
17/27 ... yes
21/29 ... yes
21/30 ... yes
21/31 ... yes
21/32 ... yes
21/33 ... yes
20/37 ... yes
13/39 ... yes
17/40 ... yes
18/41 ... yes
18/42 ... yes
10/43 ... yes
10/44 ... yes
9/45 ... yes
8/46 ... yes
17/47 ... yes
17/48 ... yes
21/49 ... yes
21/50 ... yes
21/51 ... yes
18/52 ... yes
20/53 ... yes
10/54 ... yes
42/55 ... yes
12/56 ... yes
43/57 ... yes
12/58 ... yes
21/59 ... yes
12/60 ... yes
30/61 ... yes
47/62 ... yes
47/63 ... yes
47/64 ... yes
48/65 ... yes
41/66 ... yes
48/67 ... yes
18/68 ... yes
30/69 ... yes
51/71 ... yes
20/72 ... yes
20/73 ... yes
52/74 ... yes
52/75 ... yes
54/76 ... yes
66/77 ... yes
56/78 ... yes
30/79 ... yes
30/80 ... yes
30/81 ... yes
29/82 ... yes
58/83 ... yes
12/84 ... yes
18/85 ... yes
30/86 ... yes
67/87 ... yes
41/88 ... yes
67/89 ... yes
6/90 ... yes
65/91 ... yes
56/92 ... yes
70/95 ... yes
27/96 ... yes
72/97 ... yes
30/98 ... yes
56/99 ... yes
65/100 ... yes
79/101 ... yes
73/102 ... yes
73/103 ... yes
73/104 ... yes
73/105 ... yes
65/106 ... yes
65/107 ... yes
115/108 ... yes
65/109 ... yes
17/110 ... yes
120/112 ... yes
66/113 ... yes
73/114 ... yes
90/115 ... yes
17/117 ... yes
30/118 ... yes
115/119 ... yes
115/120 ... yes
42/121 ... yes
115/122 ... yes
120/123 ... yes
120/124 ... yes
120/125 ... yes
120/126 ... yes
65/127 ... yes
73/128 ... yes
131/129 ... yes
18/130 ... yes
133/131 ... yes
66/132 ... yes
134/133 ... yes
21/134 ... yes
73/136 ... yes
25/137 ... yes
150/138 ... yes
29/139 ... yes
29/140 ... yes
30/141 ... yes
30/142 ... yes
65/143 ... yes
65/144 ... yes
65/145 ... yes
65/146 ... yes
29/147 ... yes
65/148 ... yes
174/149 ... yes
27/150 ... yes
27/151 ... yes
170/152 ... yes
132/153 ... yes
29/154 ... yes
173/155 ... yes
30/156 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.31.0 ? ... yes (2.32.0)
Git user has default SSH configuration? ... yes
Active users: ... 94
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

I kind of got it working by changing the controller https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/groups/dependency_proxy_for_containers_controller.rb#L95 to:

  def ensure_token_granted!
    result = DependencyProxy::RequestTokenService.new(image).execute

    if result[:status] == :success
      @token = result[:token]
    end
  end

And changing the code from https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/dependency_proxy/find_or_create_manifest_service.rb#L46 to something like:

    def cached_manifest_matches?(head_result)
      return true if head_result[:status] == :error && @manifest
      return false if head_result[:status] == :error

      @manifest && @manifest.digest == head_result[:digest] && @manifest.content_type == head_result[:content_type]
    end

This however seems to be a bit ugly as if it fails to get a token it still tries to call the docker hub api without any token which I think might fail. I guess it will fall back to providing the cached version in this case but it shouldn't even call it.

Edited by Wolfspirit