Container Registry multi-level image names support

What does this MR do?

Adds support for multiple container images under the same project/group namespace.

Ex: registry.example.com/group/project/any-image-name:optional-tag

Why was this MR needed?

Currently a project's registry can only host a single container image with multiple tags.

While this is appropriate for most use cases and is in line with the 12-factor principles, some valid approaches depend on multiple images. Also, the expected behavior of a Registry is to enable multi-level image names.

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

• CHANGELOG entry added
• Documentation created/updated
• API support added
• Tests
  • Added for this feature/bug
  • All builds are passing
• Conform by the merge request performance guides
• Conform by the style guides
• Branch has no merge conflicts with master (if it does - rebase it please)
• Squashed related commits together

What are the relevant issue numbers?

#17801

Marked the task CHANGELOG entry added as completed

Marked the task CHANGELOG entry added as incomplete

Added 1 commit:

• 55fad387 - Added MR Road map

Compare with previous version

Currently, no image/tags information is persisted on Gitlab. The Gitlab Container Registry client is responsible for fetching all tags for a given project through the Registry API /v2/<namespace>/tags/list endpoint.

Trying to do the same for images would imply on using the /v2/_catalog endpoint that returns every single image on the registry. The disadvantages of this approach are clear as images from other projects would have to be filtered out from the fetched results.

Another option is to persist the project's image names, which makes image enumeration and tag listing really simple. On the other hand, detecting image creation/updates from the docker CLI would be a completely new problem as these requests are proxied directly to the Docker Registry.

Are there other options? What do you think? @ayufan, @markpundsack

Added 1 commit:

• 6183a819 - Added MR Road map

Compare with previous version

Added ~19173 ~164274 ~343072 labels

I don't really see option other then storing a container images repository requested.

We can do that when you do JWT push auth to store this relation.

So lets assume that you would: docker push registry.gitlab.com/group/project/image:latest. We could store group/project/image as a place where the images can be stored.

I generally think that what would be better is to implement events executed by Docker Registry when action on Registry is executed and this information would be deliver to GitLab and stored as event.

@ayufan Agreed, the Docker Registry event endpoints can really help with this. As for persisting image paths, what do you think about storing the image name alone and then reconstruct the path like "#{group_name}/#{project_name}/#{image_name}"?

I have a couple more questions:

• To receive the events sent by Docker Registry, is creating an API endpoint the right way?
• And about this event implementation, should we store all incoming events or just process and discard them?

then reconstruct the path like

At first, I thought just store #{image_name}, but then I actually do prefer also to store full path as this is external system to us.

How about having: name and path?

path = "#{project.namespace.path}/#{project.path}/#{name}"

We are then also do ensure consitency of stored data.

To receive the events sent by Docker Registry, is creating an API endpoint the right way?

Yes

And about this event implementation, should we store all incoming events or just process and discard them?

In long-term, yes. In short-term I believe that it should be probably fine to discard them. I assumed that it would be really cool to show all registry pushed in activty feed of GitLab.

Added 176 commits:

• 6183a819...e2e3f196 - 174 commits from branch gitlab-org:master
• 52cb7944 - Added MR Road map
• de31e99e - Multi-level container image names backend implementation

Compare with previous version

Added 1 commit:

• ecb8dcc0 - Multi-level container image names backend implementation

Compare with previous version

@ayufan I just updated the branch with my first approach to the problem. As suggested, it uses the API to receive the Docker Registry notifications, enables pull/push to both #{project.path_with_namespace} and #{project.path_with_namespace}/#{container_image.name}, creates container images upon receiving a Docker Registry push event. Both container_images_repository and container_image are new models to store needed information about the container images.

In order to support an "empty image name" simulating what the Container Registry currently does some extra care was needed, this can bring some problems in the future if a new feature (e.g. nested groups) changes de amount of namespaces of a project. The ideal would be to transfer all current group/project images to the new group/project/image_name scope but I don't know how feasible this is.

I would really appreciate a review to see if the direction I am taking is in line with what you think is best

This is a sample of the registry notification endpoint from Docker Registry config.yml close to what I am using:

notifications:
  endpoints:
    - name: registry-listener
      url: https://gitlab.example.com/api/v3/registry_events
      headers:
        Content-Type: []
      timeout: 500ms
      threshold: 5
      backoff: 1s

Added 1 commit:

• 2a0c9e22 - Multi-level container image names backend implementation

Compare with previous version

Added 1 commit:

• 08335d3d - Multi-level container image names backend implementation

Compare with previous version

Resolved all discussions

@andrebsguedes

I like the approach and it makes sense for me to do it that way.

There's a couple of places that needs to be fixed:

• deletion of project and deletion of images and all tags,
• view of images and tags,
• do we need container_images_repositories?
• registry_events endpoint authorization.

@ayufan How should the endpoint authorization be done?

The Docker Registry provides a way to add a bearer token to the authentication header but this is done in the config file. Should then a single token be generated with admin rights to be able to add images to all repositories and only add that to the config file?

Added 1573 commits:

• 08335d3d...c52f3d1c - 1570 commits from branch gitlab-org:master
• b285dfb0 - Added MR Road map
• d800e593 - Multi-level container image names backend implementation
• 4923e3c6 - First iteration of conatainer_image view

Compare with previous version

@ayufan I just pushed new changes, these are some of them:

• Removed container_images_repository
• Removed redundant services
• Adapted view to display container_images as well as tags
• Fixed project, container_image and tag removal

Added 1 commit:

• 38826932 - First iteration of container_image view

Compare with previous version

Added 1 commit:

• 41622a0e - First iteration of container_image view

Compare with previous version

Marked the task Conform by the merge request performance guides as completed

Marked the task Branch has no merge conflicts with master (if it does - rebase it please) as completed

Mentioned in issue #25020 (moved)

Resolved all discussions

Reassigned to @ayufan

@andrebsguedes

Sorry for not reviewing it recently. I'll do it later and follow up you what I see next for this one :)

@ayufan No problem!

I had a busy week also, as soon as I get some time I will work on the registry authorization.

added 864 commits

• 41622a0e...632450a4 - 860 commits from branch gitlab-org:master
• 201ce168 - Added MR Road map
• ee62bb95 - Multi-level container image names backend implementation
• d4c9b530 - First iteration of container_image view
• 8bef5a67 - Adding registry endpoint authorization

Compare with previous version

Just pushed some new changes!

• Added registry token authorization
• Created Registry tab in Admin Area to view and reset the registry token
• Updated the docs to reflect new changes

/cc @ayufan

added 197 commits

• 8bef5a67...49a70d1e - 192 commits from branch gitlab-org:master
• 00f5e7f5 - Added MR Road map
• 4b1d392f - Multi-level container image names backend implementation
• dd54d448 - First iteration of container_image view
• e8585e28 - Adding registry endpoint authorization
• c30181ea - Fixes broken and missing tests

Compare with previous version

Some more changes!

• Fixed lots of broken tests
• Added some relevant tests
• Fixed wrong blob get request

/cc @ayufan

added 1 commit

• 7dd03e3e - Fixes broken and missing tests

Compare with previous version

added 1 commit

• dd908bd6 - Fixes broken and missing tests

Compare with previous version

added 1 commit

• cdc4a50c - Fixing typos in docs

Compare with previous version

added 1 commit

• d978a8f0 - Fixing typos in docs

Compare with previous version

added 1 commit

• fde8c937 - Adding mock for full_access_token

Compare with previous version

added 235 commits

• fde8c937...8f5cf205 - 228 commits from branch gitlab-org:master
• 39182a7b - Added MR Road map
• 7fc050c0 - Multi-level container image names backend implementation
• 6ff979a2 - First iteration of container_image view
• 69da98f9 - Adding registry endpoint authorization
• 967b66a9 - Fixes broken and missing tests
• 4e22be75 - Fixing typos in docs
• e56d204a - Adding mock for full_access_token

Compare with previous version

added 1 commit

• b4b58a39 - Adding mock for full_access_token

Compare with previous version

marked the task Added for this feature/bug as completed

marked the task All builds are passing as completed

marked the task Documentation created/updated as completed

marked the task Conform by the style guides as completed

@ayufan I managed to fix all build failures.

I think we need a bit of an overhaul in the view. It currently looks like this:

I just don't know if it should be done here or in a separate MR. What do you think?

Also, is there anything else I should worry about?

We still need to decide if it is worthwhile to move all images to the new naming pattern (group/project/image:tag) or if it is better to support both the new and the old one (group/project:tag). The current implementation supports both, but there are a few things that may have to be changed if the namespace pattern changes in the future (e.g. nested groups)