GitLab incorrectly deletes tags when external registry storage deletion is disabled
Summary
When deleting tags from an external registry that has storage delete disabled, the tag is removed from GitLab even though it is not removed from the underlying registry.
After refreshing, GitLab reports that "This image has no active tags" and you can see a red "Something went wrong while fetching the registry list" notification banner appear on the top of the page. See screenshot:
This apparently results in the container registry tags being left in a funky state. The customer's workaround was to enable registry storage deletion and manually trigger tag deletion via API to get the tags to be completely deleted.
This was reported (Zendesk, internal use only) by a 500-seat ultimate customer.
Steps to reproduce
- Configure an external registry with GitLab
- Set the external registry to have storage delete disabled
- Delete a tag from the container registry UI
Example Project
N/A
What is the current bug behavior?
The tag is removed from GitLab.
What is the expected correct behavior?
The tag should not be removed from GitLab.
Relevant logs and/or screenshots
Logs from deleting:
{"method":"DELETE","path":"/root/container-registry/registry/repository/1/tags/latest","format":"json","controller":"Projects::Registry::TagsController","action":"destroy","status":500,"error":"NameError: undefined local variable or method `bad_request' for #<Projects::Registry::TagsController:0x00007f66f3ba0a70>\nDid you mean? @_request","duration":8351.52,"view":0.0,"db":33.92,"time":"2019-11-11T07:57:40.790Z","params":[{"key":"namespace_id","value":"root"},{"key":"project_id","value":"container-registry"},{"key":"repository_id","value":"1"},{"key":"id","value":"latest"}],"remote_ip":"172.22.0.1","user_id":1,"username":"root","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36","queue_duration":6.3,"correlation_id":"C8JRb4y0Eza","cpu_s":5.81407651}
NameError (undefined local variable or method `bad_request' for #<Projects::Registry::TagsController:0x00007f66f3ba0a70>
Did you mean? @_request):
app/controllers/projects/application_controller.rb:65:in `method_missing'
app/controllers/projects/registry/tags_controller.rb:29:in `block (2 levels) in destroy'
app/controllers/projects/registry/tags_controller.rb:28:in `destroy'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:46:in `set_current_ip_address'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:463:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:457:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:62:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
ee/lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:48:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:32:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Logs from viewing tag list:
{"method":"GET","path":"/root/container-registry/registry/repository/1/tags","format":"json","controller":"Projects::Registry::TagsController","action":"index","status":500,"error":"NoMethodError: undefined method `map' for nil:NilClass","duration":146.1,"view":0.0,"db":22.17,"time":"2019-11-11T08:11:04.997Z","params":[{"key":"namespace_id","value":"root"},{"key":"project_id","value":"container-registry"},{"key":"repository_id","value":"1"}],"remote_ip":"172.22.0.1","user_id":1,"username":"root","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36","queue_duration":4.82,"correlation_id":"8uikEl2Uaw","cpu_s":0.0757723109999997}
NoMethodError (undefined method `map' for nil:NilClass):
lib/container_registry/tag.rb:95:in `block in layers'
lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
lib/container_registry/tag.rb:92:in `layers'
lib/container_registry/tag.rb:107:in `total_size'
app/serializers/base_serializer.rb:16:in `represent'
app/serializers/container_tags_serializer.rb:17:in `represent'
app/controllers/projects/registry/tags_controller.rb:17:in `block (2 levels) in index'
app/controllers/projects/registry/tags_controller.rb:12:in `index'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:46:in `set_current_ip_address'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:463:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:457:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:62:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
ee/lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:48:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:32:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)