Skip to content

Dependency Proxy broken after server change with backup/restore

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

The dependency proxy is broken for images, which were stored before the backup

Steps to reproduce

  1. Setup a project which uses the dependency proxy
  2. Run a pipeline (or pull an image otherwise from the proxy)
  3. Create a backup gitlab-backup create
  4. Install GitLab on another server
  5. Restore the secrets file and configuration
  6. Restore the backup gitlab-backup restore BACKUP=<name>
  7. Try to pull the already stored image from the proxy
  8. See it failing with an Exception in the logs

Example Project

Not possible to reproduce on gitlab.com as it involves moving GitLab from one server to another

What is the current bug behavior?

The pull of the image results in 500

What is the expected correct behavior?

The pull is successful with the already stored image

Relevant logs and/or screenshots

Started GET "/v2/<group>/dependency_proxy/containers/alpine/manifests/sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3" for <ip> at 2022-03-04 23:14:14 +0100
Processing by Groups::DependencyProxyForContainersController#manifest as JSON
  Parameters: {"group_id"=>"<group>", "image"=>"alpine", "tag"=>"sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3"}
Sent file /var/opt/gitlab/gitlab-rails/shared/dependency_proxy/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683/dependency_proxy/6/files/3/alpine:latest.json (0.9ms)
Completed 500 Internal Server Error in 884ms (ActiveRecord: 8.3ms | Elasticsearch: 0.0ms | Allocations: 8471)
{"method":"GET","path":"/v2/<group>/dependency_proxy/containers/alpine/manifests/sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3","format":"json","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":500,"time":"2022-03-04T22:14:15.554Z","params":[{"key":"group_id","value":"<group>"},{"key":"image","value":"alpine"},{"key":"tag","value":"sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3"}],"correlation_id":"01FXBFYVBAQA3DJ4D0GXNGHHTN","meta.user":"Taucher2003","meta.root_namespace":"<group>","meta.subscription_plan":"default","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"<ip>","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"2a0e:6a80:3:29::","user_id":4,"username":"Taucher2003","ua":"docker/20.10.12 go/go1.16.12 git-commit/459d0df kernel/4.19.0-18-amd64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)","queue_duration_s":0.025464,"request_urgency":"default","target_duration_s":1,"cf_ray":"6e6df6b56da59bd7-FRA","redis_calls":3,"redis_duration_s":0.0014810000000000001,"redis_read_bytes":7,"redis_write_bytes":279,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000526,"redis_shared_state_write_bytes":147,"redis_rate_limiting_calls":2,"redis_rate_limiting_duration_s":0.000955,"redis_rate_limiting_read_bytes":7,"redis_rate_limiting_write_bytes":132,"db_count":10,"db_write_count":0,"db_cached_count":2,"db_replica_count":0,"db_primary_count":10,"db_replica_cached_count":0,"db_primary_cached_count":2,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.011,"external_http_count":2,"external_http_duration_s":0.26040599728003144,"rack_attack_redis_count":1,"rack_attack_redis_duration_s":0.0015999572351574898,"cpu_s":0.10893,"mem_objects":12129,"mem_bytes":1066192,"mem_mallocs":2899,"mem_total_bytes":1551352,"pid":46543,"exception.class":"ActionController::MissingFile","exception.message":"Cannot read file /var/opt/gitlab/gitlab-rails/shared/dependency_proxy/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683/dependency_proxy/6/files/3/alpine:latest.json","exception.backtrace":["app/controllers/concerns/send_file_upload.rb:28:in `send_upload'","app/controllers/groups/dependency_proxy_for_containers_controller.rb:110:in `send_manifest'","app/controllers/groups/dependency_proxy_for_containers_controller.rb:26:in `manifest'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/app/controllers/ee/application_controller.rb:46:in `set_current_ip_address'","app/controllers/application_controller.rb:490:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:481:in `set_session_storage'","lib/gitlab/i18n.rb:105:in `with_locale'","lib/gitlab/i18n.rb:111:in `with_user_locale'","app/controllers/application_controller.rb:475:in `set_locale'","app/controllers/application_controller.rb:469:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:26:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"db_duration_s":0.0083,"view_duration_s":0.0,"duration_s":0.88442}
ActionController::MissingFile (Cannot read file /var/opt/gitlab/gitlab-rails/shared/dependency_proxy/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683/dependency_proxy/6/files/3/alpine:latest.json):
  
app/controllers/concerns/send_file_upload.rb:28:in `send_upload'
app/controllers/groups/dependency_proxy_for_containers_controller.rb:110:in `send_manifest'
app/controllers/groups/dependency_proxy_for_containers_controller.rb:26:in `manifest'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:46:in `set_current_ip_address'
app/controllers/application_controller.rb:490:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:481:in `set_session_storage'
lib/gitlab/i18n.rb:105:in `with_locale'
lib/gitlab/i18n.rb:111:in `with_user_locale'
app/controllers/application_controller.rb:475:in `set_locale'
app/controllers/application_controller.rb:469:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Debian 10
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.5p203
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.16
Git Version:    2.33.1.
Sidekiq Version:6.3.1
Go Version:     unknown

GitLab information
Version:        14.7.0-ee
Revision:       621e5984888
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://
HTTP Clone URL: https:///some-group/some-project.git
SSH Clone URL:  git@ssh.:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        13.22.2
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.22.2 ? ... OK (13.22.2)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
6/3 ... yes
6/4 ... yes
7/5 ... yes
6/7 ... yes
6/8 ... yes
6/9 ... yes
6/10 ... yes
6/14 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git version >= 2.33.0 ? ... yes (2.33.1)
Git user has default SSH configuration? ... yes
Active users: ... 5
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Edited by 🤖 GitLab Bot 🤖