Skip to content

Improve performance of the Container Registry delete tags API

João Pereira requested to merge 31832-improve-performance-of-the-container-registry-delete-tags-api into master

What does this MR do?

This MR addresses #31832 (closed) (and #34510 (closed) as a side effect) by adding support to delete tags by name instead of digest.

These changes depend on the new tag delete API route implemented in our Container Registry fork (container-registry!51 (merged)). The current workaround (delete tags by digest, implemented in !16886 (merged)) is used as a fallback option to maintain compatibility with other registries (or a version of our registry fork before container-registry!51 (merged)).

Notes

  • New features only supported by the GitLab registry fork are becoming more common (as this one) and the necessity to support other registries requires workarounds and fallback options. We have created #199117 to discuss the long term compatibility strategy for third-party registries. Until a conclusion is reached, it's better not to make any deep refactoring around the integration with multiple registries (e.g. a registry abstraction layer).

  • These changes are scoped by #31832 (closed). Therefore we have only implemented the minimum required changes to improve performance for the frontend/API, in Project::ContainerRepository::DeleteTagsService. Once this MR is merged, and preferably after #199117 is resolved, the same improvements can be applied to Projects::ContainerRepository::CleanupTagsService and Geo::ContainerRepositorySync as well.

  • Although the performance improvements implemented in this MR depend on a new registry API route, this can be deployed independently, as GitLab will use the fallback behaviour (delete by digest) whenever the registry doesn't support tag deletion.

Rationale

As we're only changing the underlying behaviour (the communication between GitLab and the Container Registry), the GitLab API and UI functionality remain unchanged.

Current Implementation

We currently have to make the following network requests against the Container Registry to delete a tag without causing the soft deletion of the associated manifest and other related tags as well:

  1. Create a dummy blob;
  2. Upload the blob to the registry;
  3. Create a manifest, associating the blob with the tag to delete. This step is done N times for bulk requests, where N is the number of tags to delete;
  4. Delete the manifest created in the previous step.

For more details, please see #15737 (closed) and !16886 (merged). This was the best possible workaround back then, as the registry API had no support to delete tags.

Considering the above, in the worst-case scenario, to delete N tags we currently need 3+N network requests.

Proposal

Only one network request is needed to delete a tag with the change proposed on this MR.

The request is a DELETE /v2/<name>/tags/reference/<reference>, where <name> is the repository name (e.g. gitlab-org/gitlab-test) and <reference> is the tag name (e.g. 1.0.0).

The responsibility of ensuring that the referenced manifest and other tags are not soft-deleted is offloaded to the Container Registry.

Therefore, to delete N tags we need N network requests against the Container Registry.

To maintain compatibility with third-party registries that don't support tag deletion, the registry client (ContainerRegistry::Client) performs a single OPTIONS /v2/<name>/tags/reference/<reference> request against the registry. If the response Allow header includes DELETE, then we know that the registry supports tag deletion. Otherwise we fallback to deleting tags by digest. Once #199117 is resolved, we need to consider a static way of determining whether the registry being used is the GitLab fork or not, avoiding this OPTIONS request. #204839 (closed) was raised to track this.

This new behaviour is behind a new feature flag, named container_registry_fast_tag_delete, which defaults to true.

Demonstration

Here we show the process of deleting tags, comparing the current implementation with the proposal on this MR.

Setup

  1. Setup a GDK instance (instructions) using this branch and including the Container Registry integration (instructions). We'll assume that the GitLab web application is listening at 198.18.0.1:3000.
  2. In a separate tab, tail the container registry logs with the command gdk tail registry.
  3. We'll use the Gitlab Org/GitLab Test sample project for this demonstration. Head to http://198.18.0.1:3000/gitlab-org/gitlab-test/container_registry and make sure there are no container images stored for this project.
  4. Upload a random image with three tags, all pointing to the same manifest. We'll assume that the REGISTRY_ADDR environment variable was set to the Container Registry address (host:port): 
    for i in {1..3}
do 
  docker tag alpine:latest $REGISTRY_ADDR/gitlab-org/gitlab-test:$i
  docker push $REGISTRY_ADDR/gitlab-org/gitlab-test:$i
done
  5. Refresh the container registry page in GitLab. We should see a single image with three tags, all with the same Image ID.
  6. Open the browser development tools. We'll be looking at the network requests logs (optional).

Current Implementation (fallback)

For this test case, we use the default Container Registry image (registry:2). No further changes required. As the default registry image doesn't implement the new tag delete API, the GitLab application will fallback to deleting tags by digest.

Delete

  1. Head to http://198.18.0.1:3000/gitlab-org/gitlab-test/container_registry. List the available tags for gitlab-org/gitlab-test and delete one of them. We'll remove tag 1.
  2. Looking at the Container Registry logs, we can see (truncated): 
    2020-02-10_10:56:50.92323 registry              : 172.17.0.1 - - [10/Feb/2020:10:56:50 +0000] "OPTIONS /v2/name/tags/reference/tag HTTP/1.1" 404 19 "" "Faraday v0.15.4"
2020-02-10_10:56:50.93520 registry              : 172.17.0.1 - - [10/Feb/2020:10:56:50 +0000] "POST /v2/gitlab-org/gitlab-test/blobs/uploads/ HTTP/1.1" 202 0 "" "Faraday v0.15.4"
2020-02-10_10:56:50.95791 registry              : 172.17.0.1 - - [10/Feb/2020:10:56:50 +0000] "PUT /v2/gitlab-org/gitlab-test/blobs/uploads/1830a77e-6b80-4ad8-b9e0-38947e6bfff2?_state=cQ8M09KQw1KpFykXnLMo2u-qagcXKyhwJ8mZfIwc_j17Ik5hbWUiOiJnaXRsYWItb3JnL2dpdGxhYi10ZXN0IiwiVVVJRCI6IjE4MzBhNzdlLTZiODAtNGFkOC1iOWUwLTM4OTQ3ZTZiZmZmMiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wMi0xMFQxMDo1Njo1MC45MzY2NjI0WiJ9&digest=sha256%3A4435000728ee66e6a80e55637fc22725c256b61de344a2ecdeaac6bdb36e8bc3 HTTP/1.1" 201 0 "" "Faraday v0.15.4"
2020-02-10_10:56:50.96997 registry              : 172.17.0.1 - - [10/Feb/2020:10:56:50 +0000] "PUT /v2/gitlab-org/gitlab-test/manifests/1 HTTP/1.1" 201 0 "" "Faraday v0.15.4"
2020-02-10_10:56:51.00603 registry              : 172.17.0.1 - - [10/Feb/2020:10:56:50 +0000] "DELETE /v2/gitlab-org/gitlab-test/manifests/sha256:7bdb5df100f9b601290cb6d72fbeb5b3aa73f4908fe1d8075b00df013c79c43c HTTP/1.1" 202 0 "" "Faraday v0.15.4"
    The logs show four requests to delete tag 1:
    1. Blob creation;
    2. Blob upload;
    3. Manifest 1 creation;
    4. Manifest deletion.

We can also see that these requests were preceded by a OPTIONS /v2/name/tags/reference/tag request to check whether the registry supports tag deletion or not. In this case it doesn't, so a 404 Not Found response is returned.

Bulk Delete

  1. Select the remaining two tags, 2 and 3, and remove them in bulk.
  2. Looking at the Container Registry logs, we can see (truncated): 
    2020-02-10_11:00:35.53353 registry              : 172.17.0.1 - - [10/Feb/2020:11:00:35 +0000] "POST /v2/gitlab-org/gitlab-test/blobs/uploads/ HTTP/1.1" 202 0 "" "Faraday v0.15.4"
2020-02-10_11:00:35.55880 registry              : 172.17.0.1 - - [10/Feb/2020:11:00:35 +0000] "PUT /v2/gitlab-org/gitlab-test/blobs/uploads/e65cf2fb-204e-41e1-a9cd-d72813d14c1e?_state=brmk9Le5Oca_hRSMeqkkKV8ZdI7eo98wrDPp162EH4t7Ik5hbWUiOiJnaXRsYWItb3JnL2dpdGxhYi10ZXN0IiwiVVVJRCI6ImU2NWNmMmZiLTIwNGUtNDFlMS1hOWNkLWQ3MjgxM2QxNGMxZSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wMi0xMFQxMTowMDozNS41NTA4NDI0WiJ9&digest=sha256%3A4435000728ee66e6a80e55637fc22725c256b61de344a2ecdeaac6bdb36e8bc3 HTTP/1.1" 201 0 "" "Faraday v0.15.4"
2020-02-10_11:00:35.57171 registry              : 172.17.0.1 - - [10/Feb/2020:11:00:35 +0000] "PUT /v2/gitlab-org/gitlab-test/manifests/2 HTTP/1.1" 201 0 "" "Faraday v0.15.4"
2020-02-10_11:00:35.58437 registry              : 172.17.0.1 - - [10/Feb/2020:11:00:35 +0000] "PUT /v2/gitlab-org/gitlab-test/manifests/3 HTTP/1.1" 201 0 "" "Faraday v0.15.4"
2020-02-10_11:00:35.61928 registry              : 172.17.0.1 - - [10/Feb/2020:11:00:35 +0000] "DELETE /v2/gitlab-org/gitlab-test/manifests/sha256:7bdb5df100f9b601290cb6d72fbeb5b3aa73f4908fe1d8075b00df013c79c43c HTTP/1.1" 202 0 "" "Faraday v0.15.4"
    The logs show five requests to delete tags 2 and 3:
    1. Blob creation;
    2. Blob upload;
    3. Manifest 2 creation;
    4. Manifest 3 creation;
    5. Manifest deletion.

Proposal

For this test case we use a custom image for the Container Registry based on the new-tags-api branch.

Use the following commands to build the custom image and reconfigure GDK (reference):

git clone --single-branch --branch new-tags-api git@gitlab.com:gitlab-org/container-registry.git
cd container-registry
docker build -t registry:new-tags-api .
cd $GDK_PATH
echo registry:new-tags-api > registry_image
gdk reconfigure
gdk restart

Confirming that we're using the correct image:

$ docker ps
CONTAINER ID        IMAGE                   ...
6b1522686d27        registry:new-tags-api   ...

Delete

  1. Head to http://198.18.0.1:3000/gitlab-org/gitlab-test/container_registry.
  2. List the available tags for gitlab-org/gitlab-test:
  3. Delete tag 1.
  4. Looking at the Container Registry logs we can see a single DELETE request (note that the OPTIONS request succeeded this time):
2020-02-10_11:36:24.81259 registry              : 172.17.0.1 - - [10/Feb/2020:11:36:24 +0000] "OPTIONS /v2/name/tags/reference/tag HTTP/1.1" 200 0 "" "Faraday v0.15.4"
2020-02-10_11:36:24.83128 registry              : 172.17.0.1 - - [10/Feb/2020:11:36:24 +0000] "DELETE /v2/gitlab-org/gitlab-test/tags/reference/1 HTTP/1.1" 202 0 "" "Faraday v0.15.4"
  1. Although they reference the same manifest, tags 2 and 3 were not deleted and remain on the list: Screenshot_2020-01-22_at_12.55.09

Bulk Delete

  1. Select the remaining two tags, 2 and 3, and remove them in bulk. The list should now be empty: Screenshot_2020-01-22_at_12.57.37
  2. Looking at the Container Registry logs we can see two DELETE requests:
2020-02-10_11:36:56.32980 registry              : 172.17.0.1 - - [10/Feb/2020:11:36:56 +0000] "DELETE /v2/gitlab-org/gitlab-test/tags/reference/2 HTTP/1.1" 202 0 "" "Faraday v0.15.4"
2020-02-10_11:36:56.34684 registry              : 172.17.0.1 - - [10/Feb/2020:11:36:56 +0000] "DELETE /v2/gitlab-org/gitlab-test/tags/reference/3 HTTP/1.1" 202 0 "" "Faraday v0.15.4"

Performance

Number of Requests

As described above, in the worst-case scenario, the current implementation requires 3+N network requests to delete N tags while the proposed change requires only N. Therefore, the proposed implementation requires at least three fewer network requests per tag to delete.

Throughput

Although the difference between the current implementation and this MR is not exceptional in terms of the number of required network requests, the throughput is a different subject.

As described above, the current workaround requires the creation and upload of a dummy blob and manifest(s), whose requests require considerably more time than a simple delete.

Test Methodology

To test throughput, we have created a large number of tags (adapting the script listed in Setup) and then deleted 1 tag and 10 tags (bulk delete) at a time through the UI.

Additionally, the limit of tags per bulk delete was raised temporarily from 15 to 20 (here), so that we can see how the improvement develops when we double the number of tags from 10 to 20. We also enabled config.cache_classes before performing the tests.

Each test was run three times, and the average elapsed time (in milliseconds) was calculated. The elapsed time was measured using the browser (Firefox) Web Console by isolating each DELETE request in the timeline.

Test Results

Number of tags deleted (N) Delete by digest (ms) Delete by name (ms) Improvement (%)
1 367 121 67%
10 (bulk) 699 297 58%
20 (bulk) 1030 503 51%

Considering the results above, the proposed approach in this MR is up to ~60% faster than the current implementation. Deleting 20 tags is now faster than it was to delete 10.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • ~~Label as security and @ mention @gitlab-com/gl-security/appsec~~
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by João Pereira

Merge request reports