Mirror Docker Registry's structures in database
This issue covers some part of https://gitlab.com/gitlab-org/gitlab-ee/issues/2870 and https://gitlab.com/gitlab-org/gitlab-ce/issues/29639
We had a call with @ayufan and agreed that we need to track not only tag names but exact image references too, something like:
container_repositories - project_id - name container_tags - repository_id - name container_tag_versions - tag_id - digest - size - layers (int) --- tag = repository.create_or_update_tag(name: :latest) tag.create_version(digest: AA, size: BB, layers: CC)
The thing is that currently, Docker Registry API does not provide any API endpoint to see the list of all images, we can only see the list of tags. That means that if you updated some tag, the old image is still accessible and we need to replicate it to secondary too. By listening to registry events we can create records in a database and so we'll have a number of benefits, like quick access to the actual registry state, ability to replicate images to secondary nodes and other.
This issue implies few subtasks:
- Create shared secret between GitLab app and Docker Registry so we can authenticate the notification calls.
Support for this secret on omnibus and gdk side (two separate MRs)
- Create models and API endpoints to handle Docker Registry event envelopes.
- Create a mechanism that will initialize the data in those tables. We can use API to get information on currently available repositories and to get a list of tags.
- Implement code that will allow existing registry's consumers to consume from data from both sources, legacy (always using registry API) and a new one (from database and API if needed). We need it because the migration process will take some time and it should be smooth.