Gitlab keeps tryong to delete non-existing container images
Summary
Gitlab tries to delete container images from the registry that already have been garbage collected, which causes a lot of error log messages (almost 800k/h).
Steps to reproduce
I am not sure this is how it happened, but I suspect it's the cause, because we have a cronjob
sudo docker exec -it gitlab-registry-1 registry garbage-collect -m /etc/docker/registry/config.yml --dry-run=false
.
- Push an image to the registry
- Have it garbage collected
- Enable cleanup
- Watch it fail
What is the current bug behavior?
Gitlab perpetually ties to delete already deleted images from the registry.
What is the expected correct behavior?
Gitlab should remove the non-existing image from the database, and remove it from the cleanup queue.
Relevant logs and/or screenshots
This is from the registry container
time="2024-02-06T13:31:35.125754278+01:00"
level=error
msg="response completed with error"
auth.user.name=
err.code="digest invalid"
err.message="provided digest did not match uploaded content"
go.version=go1.20.8
http.request.host="registry:5000"
http.request.id=d8e157da-1ee4-401d-b07c-b608b9870fbc
http.request.method=DELETE
http.request.remoteaddr="172.18.0.6:42308"
http.request.uri="/v2/infra/smtp/manifests/renovate-ubuntu-22-04"
http.request.useragent="GitLab/16.8.1"
http.response.contenttype="application/json;
charset=utf-8"
http.response.duration=3.651935ms
http.response.status=400
http.response.written=98
vars.name="infra/smtp"
vars.reference=renovate-ubuntu-22-04
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: root Using RVM: no Ruby Version: 3.1.4p223 Gem Version: 3.3.26 Bundler Version:2.5.4 Rake Version: 13.0.6 Redis Version: 5.0.7 Sidekiq Version:7.1.6 Go Version: unknown GitLab information Version: 16.8.1 Revision: a658d811568 Directory: /home/git/gitlab DB Adapter: PostgreSQL DB Version: 16.1 URL: https://gitlab.repository.corp.gdaag.de HTTP Clone URL: https://gitlab.repository.corp.gdaag.de/some-group/some-project.git SSH Clone URL: ssh://git@gitlab.repository.corp.gdaag.de:10022/some-group/some-project.git Using LDAP: yes Using Omniauth: no GitLab Shell Version: 14.33.0 Repository storages: - default: unix:/home/git/gitlab/tmp/sockets/private/gitaly.socket GitLab Shell path: /home/git/gitlab-shell Gitaly - default Address: unix:/home/git/gitlab/tmp/sockets/private/gitaly.socket - default Version: 16.8.1 - default Git Version: 2.42.0
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.33.0 ? ... OK (14.33.0) Running /home/git/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 37 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... yes Systemd unit files or init script up-to-date? ... yes Projects have namespace: ... 6/3 ... yes [all other "yes" lines ommmitted] 680/853 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 3.0.6 ? ... yes (3.1.4) Git user has default SSH configuration? ... yes Active users: ... 52 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished