Cleanup tags service: cache created_at values for tags that are kept (!69459) · Merge requests · GitLab.org / GitLab

David Fernandez requested to merge 339129-cleanup-policies-caching into master Sep 02, 2021

⚾ Context

The container registry allows users to store container images and related tags. With time, tags tend to be accumulated.

To counter that, we implemented cleanup policies. Those are basically a set of filters that describe which tags can be destroyed and which ones need to be kept. Cleanup policies are executed by background jobs. Those jobs are quite simple. Load the policy, load the tags from the given container image, run the filters and for all the tags that we don't want to keep, destroy them. Simple and easy.

Things get more challenging from the fact that container images are persisted in the database on the rails side but that's not the case for tags. To know the list of tags of a container image, we need to use the container registry API. To know details of a container image tag, we also need to use the container registry API.

This MR will focus on the cleanup tags service. It receives the filters from the container expiration policy and the container image. Its purpose is to ping the container registry API to build the list of tags to delete.

Let's have an interaction schema for this service.

sequenceDiagram
    autonumber
    rails cleanup tags service->>container registry: Give me all the tags of container image X
    container registry->>rails cleanup tags service: Array of tag names
    rails cleanup tags service->>rails cleanup tags service: Apply filters F1 on the array of tag names
    rails cleanup tags service->>rails cleanup tags service: Truncate the list
    loop **For each tag name**
      rails cleanup tags service->>container registry: get the created at timestamp (simplified, this triggers 2 network requests in reality)
      container registry->>rails cleanup tags service: return the created at timestamp
    end
    rails cleanup tags service->>rails cleanup tags service: Apply filters F2 on tags with created at
    rails cleanup tags service->>rails delete tags service: hey delete these tags!
    rails delete tags service->>rails cleanup tags service: ok

There is a new service in this interaction schema (8. and 9.). The delete tags service does what the name says. It takes a bunch of tags, loop through them and for each tag, send a DELETE request to the container registry API.

Now filters are applied in two steps: F1(3.) and F2(7.). This is because the container registry will return the list of tags of a container image as an Array of tag names and that's it. Basically, F1 are all the filters that can run on the tag name only. Applying F1 we get all the tags that can (potentially) be destroyed.

Now, we need to apply F2 but F2 are filters that are based on the tag's created timestamp. The backend doesn't have that timestamp yet. To get it, it has to (again) ping the container registry API and yes, it's one request per tag.

You can easily the problem here, we can have a quite big list here and pinging the container registry for all tags created timestamps is not feasible. To cope with this, we implemented a truncate step (4.) right before F2. This truncation will keep at most n tags for destruction. Currently on gitlab.com, we're using n = 200 (that's an application setting). So the worst case scenario, we will ping 200 times the container registry API.

Before moving one, let's zoom on F2 filters. They are applied in the following order:

keep_n: the list of tags is ordered by created (descending) and we keep the n first tags.
older_than: defines how long tags are kept. One possible value could be 10 days. That means that tags younger than 10 days will be kept by this filter.
- Here is the distribution of the possible values.
- We can see that 80%+ of the enabled policies have this value set.

Both filters are optional. Note that if set, older_than is the very last filter to remove tags from the list of tags to destroy.

All the above is for only one cleanup on one container image. As we roll out this feature to more and more projects, we increase the number of cleanups to do. For ~performance reasons, the max number of jobs that can be enqueued for those cleanups are limited (currently 7 for gitlab.com).

During the feature rollout, we noticed that we reached the limit of the current implementation. In short words, we bumped the number of jobs but that didn't bring the improvements we wanted.

So in other words, we need to improve the efficiency of the cleanup job so that it uses less time than currently.

The idea developed in #339129 (closed) is the following:

Container image tags are mutable objects. They can have a created field in one ping and on the next one, it changed.
- This should be a rare thing but for this MR, it's better to consider that tags to not be immutable.
Looking at this chart, we see that most of the times, the cleanup job ping the container registry, build the list of tags to destroy and that list is..... 🥁 empty. So, we end up with no tags to destroy.
- Those no op executions is what we want to speed up
Looking at the interaction schema above, it's easy to spot the problem. That loop to get the created_at fields costs time and network resources.
We want to cache the created_at field in redis so that we avoid unnecessary pings to the container registry.
Do we want all the created_at fields? Nop. Caching a created field for a tag that is going to be destroyed is pointless. So we're going to only cache created values for tags that are kept
How do we evict the cache entry? We can't keep the created cached forever. Remember, tags are mutable objects. older_than defines how long we keep tags. That's exactly what we're going to use as a TTL on the cache entry.
- Let's have an example
  1. t: a tag is created
  2. t+x (before older_than): cleanup runs, t is cached with a ttl of t+older_than (because the tag is kept)
  3. t+ttl: the cache entry is evicted
  4. t+ttl+y: the cleanup runs again. created of the tag is fetched fresh out of the container registry. Assuming that it didn't change, the tag is scheduled for destruction (we will no cache that created).
- The important thing to note with the example above is that the "scheduled for destruction" decision is made out of a fresh response of the container registry API. This is highly important. As we said, tags are mutable objects so we can easily have stale cache entries. We acknowledge that fact and we take care to not take the decision of removing a tag out of data coming from the cache.

Last piece of context, known that cleanup policies are run on a cadence. Meaning that we have some space here: the decision to delete a tag can be delayed (meaning that the destruction happens to execution+1 instead of execution) but once a tag is destroyed, this must be 100% be accurate (we must avoid destroying a tag that was not meant to be destroyed).

From the above is that we don't need to be 100% accurate when taking the decision to keep a tag. If it will get destroyed at a later execution, that's totally fine.

🔍 What does this MR do?

Implement caching in the cleanup tags service
- Caching will only be used under two conditions:
  1. The service is invoked as part of a cleanup policy execution
  2. The older_than filter is set. Even tough this is an optional filter, our sisense dashboard that it is used for 80%+ of the enabled policies.
Upgrade the cleanup tags service to a BaseContainerService for better readability
Update the related specs
No documentation change as this is a ~performance improvement.

🖼 Screenshots or Screencasts (strongly suggested)

See How to setup and validate locally, just below.

⚙ How to setup and validate locally (strongly suggested)

Requirements:

Have GDK up and running with the container registry enabled.
- You will need rails background jobs too.
Create a new project
Using https://gitlab.com/nmezzopera/container-factory, push 10 tags to 2 container images: $ ./publish.sh <project_path> 10 2 gdk.test:5000
Check <gitlab_base_url>/<project_path>/container_registry that your have 2 images with 10 tags each
On a console tab, let's tail the container registry logs to get the requests: $ gdk tail registry

Case: `no op` cleanups

Using a rails console, let's fine tune the cleanup policy of the project:

pry(main)> pol = ContainerExpirationPolicy.find(<project_id>)
pry(main)> pol.update!(keep_n: 5, enabled: true)
pry(main)> pol.update_column(:next_run_at, 3.seconds.ago)

Now, let's trigger a cleanup background job

pry(main)> ContainerExpirationPolicyWorker.perform_in(5.seconds)

After a while, the cleanups kicks in and each container image will still have its 10 tags each. That's a no op cleanup.

Let's check the registry logs: (after some cleaning)

registry logs

2021-09-02_13:58:01.62857 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/tags/list HTTP/1.1" 200 187 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.64431 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/alpha-22576 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.66608 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:857c886c8db76d3caa68837a2c88c01e7ffc4c121b2431f47741439fee44a456 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.68108 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/alpha-28158 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.69315 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:e6bd57840c299a8b334abc5bb15df89a371dc17efd85cd898f70abe9f5f757df HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.70858 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/alpha-8994 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.71916 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/tags/list HTTP/1.1" 200 192 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.72605 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:f97074f999ebd3b177cb6fc3c4b3c04b7d3879667974df05022c935ecc484759 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.73886 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/alpha-30192 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.74246 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/beta-17313 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.77002 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:1e4c86726e9b16a36ea1e5dc7dcf0a94d98a6178c7d17ead6c172784df23be57 HTTP/1.1" 200 1171 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.77009 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:1a3bf6b6f97e5e5e193f7a60d0bcb98430e9f8e3e20c9a1630d516c7440828cf HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.79092 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/beta-537 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.79274 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/latest-11123 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.80426 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:560822d298ba19df7c64a989dc7c0e670c72e309c83522ce931095f6fe4a86ae HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.82719 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:80375371ec09f7b05103e0e0113da0d6282c22f94263bac3e6e5e3239c3662cd HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.83804 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/beta-8542 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.84960 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/latest-15189 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.85185 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:c7cf73aaacad08b85e6a61531896af8111547af3df1bf3507ef0585a41547f5b HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.88163 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:2b85a86e5aa5d89f50486508719fcfc3ca8b37671776293d19478f6f6a2d14a2 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.88165 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/stable-13718 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.88209 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:715fc72557595a874d991c06411bbbb4bd856596f129e78362ad2372113dfc08 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.89131 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/latest-19311 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.90389 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:b80ffc12beb664efdd015e89ff7d5136ecd245969e6dae6bc16688b689dbce09 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.90494 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/stable-14948 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.91961 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:1fb2d6f41b5d7af619d99386b829dbba93e0ff8ab302cd3cd6f0c3dc1813ef85 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.92863 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/latest-9828 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.94160 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:b874977ba1309f228748a81308b25f3016449c30552363368ce5862c717f5689 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.94202 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/stable-4693 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.95446 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:529a4fd0b51fba4016f6c581cf5cb733215be30d898d3492ebfe465c9b304681 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.96240 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/next-30141 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.98322 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/stable-5443 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.98596 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:d5d7cf2e729bdf1c1ccd1e649bf2055e8f4882a6fe3e82c014f1024d9635d2c4 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:01.99962 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:08d7608c172d83914e08f333a4e37d46beffb12c1ec51673527063ce18ad791e HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:02.00468 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:01 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/next-32207 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:02.02316 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:02 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:f2aaa07eb3cb4499d5dadc4691ad2d40b71e36c8afd6a23254ab1fe8d920b773 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:02.03946 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:02 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/next-4503 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:02.05105 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:02 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:f65119399a53f7aeb47f3400a8cc0ca164beda9c321f704d5e44941a22dde7a9 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:02.06508 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:02 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/stable-21833 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:02.07730 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:02 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:48501630e050188d53fdfe63fe3e8b8d708f41e8b0993f545cc1ddd0f2407daf HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:02.09381 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:02 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/stable-25326 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_13:58:02.10493 registry              : 172.17.0.1 - - [02/Sep/2021:13:58:02 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:c499da3d4326ea6d371a1df967c58db22c4dac48c154f4da851a983430c0e4e3 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"

We have 42 requests because, for each container, we have:

1 request to get the list of tags
2 requests per tag to get the created field.

😱

Let's trigger another cleanup and check the registry logs:

pry(main)> pol.update_column(:next_run_at, 3.seconds.ago)
pry(main)> ContainerExpirationPolicyWorker.perform_in(5.seconds)

The registry logs:

2021-09-02_14:06:09.37718 registry              : 172.17.0.1 - - [02/Sep/2021:14:06:09 +0000] "GET /v2/root/clpol_caching/gitlab-27264/tags/list HTTP/1.1" 200 187 "" "GitLab/14.3.0-pre"
2021-09-02_14:06:09.48835 registry              : 172.17.0.1 - - [02/Sep/2021:14:06:09 +0000] "GET /v2/root/clpol_caching/rails-26743/tags/list HTTP/1.1" 200 192 "" "GitLab/14.3.0-pre"

We get only the 2 tags list requests! 🚀 That's because all the tags were kept in the previous cleanup = all the created values were cached in redis. We can verify this with:

(from GDK root)

Redis keys

redis-cli -s redis/redis.socket                                                                           
redis redis/redis.socket> keys container_repository
(empty array)
redis redis/redis.socket> keys container_repository*
 1) "container_repository:{5}:tag:latest-9828:created_at"
 2) "container_repository:{4}:tag:beta-537:created_at"
 3) "container_repository:{5}:tag:next-30141:created_at"
 4) "container_repository:{4}:tag:alpha-8994:created_at"
 5) "container_repository:{5}:tag:stable-25326:created_at"
 6) "container_repository:{5}:tag:latest-19311:created_at"
 7) "container_repository:{5}:tag:next-32207:created_at"
 8) "container_repository:{4}:tag:beta-8542:created_at"
 9) "container_repository:{4}:tag:alpha-22576:created_at"
10) "container_repository:{4}:tag:stable-14948:created_at"
11) "container_repository:{4}:tag:alpha-28158:created_at"
12) "container_repository:{5}:tag:next-4503:created_at"
13) "container_repository:{4}:tag:beta-17313:created_at"
14) "container_repository:{5}:tag:latest-11123:created_at"
15) "container_repository:{4}:tag:stable-4693:created_at"
16) "container_repository:{4}:tag:stable-5443:created_at"
17) "container_repository:{4}:tag:stable-13718:created_at"
18) "container_repository:{5}:tag:alpha-30192:created_at"
19) "container_repository:{5}:tag:latest-15189:created_at"
20) "container_repository:{5}:tag:stable-21833:created_at"
redis redis/redis.socket>

Here are our 20 keys corresponding to our 20 tags.

Case: Cleanup that actually removes tags

Stop gdk so that redis is reset. $ gdk stop

This is a bit more involved as we need to manually update the created field of the tags we want to be removed by the cleanup.

From the GDK root, the blobs are accessible by their sha256 in registry/storage/docker/registry/v2/blobs/sha256. In the sha256 directory, there is a data file that contains the json structure. That json is exactly what the registry returns. It's just a matter of updating the created key.

I updated two tags created_at so that they were created one year ago. I expected them to be destroyed by the cleanups and to not be cached in redis.

First cleanup:

pry(main)> pol = ContainerExpirationPolicy.find(<project_id>)
pry(main)> pol.update_column(:next_run_at, 3.seconds.ago)
pry(main)> ContainerExpirationPolicyWorker.perform_in(5.seconds)

and here is the registry logs (cleaned up):

Registry logs

2021-09-02_14:52:22.17036 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/tags/list HTTP/1.1" 200 187 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.22270 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/alpha-22576 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.29329 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:857c886c8db76d3caa68837a2c88c01e7ffc4c121b2431f47741439fee44a456 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.34122 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/tags/list HTTP/1.1" 200 192 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.36525 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/alpha-28158 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.39858 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/alpha-30192 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.39860 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:e6bd57840c299a8b334abc5bb15df89a371dc17efd85cd898f70abe9f5f757df HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.46699 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/alpha-8994 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.47125 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:1a3bf6b6f97e5e5e193f7a60d0bcb98430e9f8e3e20c9a1630d516c7440828cf HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.50595 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:f97074f999ebd3b177cb6fc3c4b3c04b7d3879667974df05022c935ecc484759 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.51873 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/latest-11123 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.55483 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/beta-17313 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.57940 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:80375371ec09f7b05103e0e0113da0d6282c22f94263bac3e6e5e3239c3662cd HTTP/1.1" 200 1174 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.59629 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:1e4c86726e9b16a36ea1e5dc7dcf0a94d98a6178c7d17ead6c172784df23be57 HTTP/1.1" 200 1171 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.62819 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/latest-15189 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.64765 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/beta-537 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.67187 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:2b85a86e5aa5d89f50486508719fcfc3ca8b37671776293d19478f6f6a2d14a2 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.69492 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:560822d298ba19df7c64a989dc7c0e670c72e309c83522ce931095f6fe4a86ae HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.72096 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/latest-19311 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.74544 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/beta-8542 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.76451 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:b80ffc12beb664efdd015e89ff7d5136ecd245969e6dae6bc16688b689dbce09 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.79271 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:c7cf73aaacad08b85e6a61531896af8111547af3df1bf3507ef0585a41547f5b HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.81122 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/latest-9828 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.84491 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/stable-13718 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.85559 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:b874977ba1309f228748a81308b25f3016449c30552363368ce5862c717f5689 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.88381 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:715fc72557595a874d991c06411bbbb4bd856596f129e78362ad2372113dfc08 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.89813 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/next-30141 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.93286 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:d5d7cf2e729bdf1c1ccd1e649bf2055e8f4882a6fe3e82c014f1024d9635d2c4 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.93419 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/stable-14948 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.97268 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:1fb2d6f41b5d7af619d99386b829dbba93e0ff8ab302cd3cd6f0c3dc1813ef85 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:22.98671 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:22 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/next-32207 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.02624 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:f2aaa07eb3cb4499d5dadc4691ad2d40b71e36c8afd6a23254ab1fe8d920b773 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.02626 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/stable-4693 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.06539 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:529a4fd0b51fba4016f6c581cf5cb733215be30d898d3492ebfe465c9b304681 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.07949 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/next-4503 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.11223 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:f65119399a53f7aeb47f3400a8cc0ca164beda9c321f704d5e44941a22dde7a9 HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.12274 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/gitlab-27264/manifests/stable-5443 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.15584 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/gitlab-27264/blobs/sha256:08d7608c172d83914e08f333a4e37d46beffb12c1ec51673527063ce18ad791e HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.16148 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/stable-21833 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.21010 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:48501630e050188d53fdfe63fe3e8b8d708f41e8b0993f545cc1ddd0f2407daf HTTP/1.1" 200 1173 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.25371 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/rails-26743/manifests/stable-25326 HTTP/1.1" 200 524 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.28807 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "GET /v2/root/clpol_caching/rails-26743/blobs/sha256:c499da3d4326ea6d371a1df967c58db22c4dac48c154f4da851a983430c0e4e3 HTTP/1.1" 200 1174 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.36795 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "DELETE /v2/root/clpol_caching/rails-26743/tags/reference/latest-11123 HTTP/1.1" 202 0 "" "GitLab/14.3.0-pre"
2021-09-02_14:52:23.41383 registry              : 172.17.0.1 - - [02/Sep/2021:14:52:23 +0000] "DELETE /v2/root/clpol_caching/rails-26743/tags/reference/stable-25326 HTTP/1.1" 202 0 "" "GitLab/14.3.0-pre"

42 requests as in the first case
2 request to DELETE the 2 tags
From this log, we get that the cleanup did what was expected: the 2 tags have been destroyed.

Let's redo a cleanup

pry(main)> pol.update_column(:next_run_at, 3.seconds.ago)
pry(main)> ContainerExpirationPolicyWorker.perform_in(5.seconds)

Let's look at the registry logs:

2021-09-02_14:55:02.41129 registry              : 172.17.0.1 - - [02/Sep/2021:14:55:02 +0000] "GET /v2/root/clpol_caching/gitlab-27264/tags/list HTTP/1.1" 200 187 "" "GitLab/14.3.0-pre"
2021-09-02_14:55:02.56525 registry              : 172.17.0.1 - - [02/Sep/2021:14:55:02 +0000] "GET /v2/root/clpol_caching/rails-26743/tags/list HTTP/1.1" 200 162 "" "GitLab/14.3.0-pre"

2 requests as expected

Let's check the redis keys: (from GDK root)

redis keys

redis-cli -s redis/redis.socket  
redis redis/redis.socket> keys container_repository*
 1) "container_repository:{5}:tag:next-4503:created_at"
 2) "container_repository:{5}:tag:latest-15189:created_at"
 3) "container_repository:{5}:tag:next-32207:created_at"
 4) "container_repository:{5}:tag:latest-9828:created_at"
 5) "container_repository:{4}:tag:beta-537:created_at"
 6) "container_repository:{5}:tag:stable-21833:created_at"
 7) "container_repository:{4}:tag:stable-14948:created_at"
 8) "container_repository:{5}:tag:alpha-30192:created_at"
 9) "container_repository:{4}:tag:beta-8542:created_at"
10) "container_repository:{4}:tag:alpha-28158:created_at"
11) "container_repository:{4}:tag:alpha-22576:created_at"
12) "container_repository:{4}:tag:alpha-8994:created_at"
13) "container_repository:{4}:tag:stable-13718:created_at"
14) "container_repository:{5}:tag:latest-19311:created_at"
15) "container_repository:{5}:tag:next-30141:created_at"
16) "container_repository:{4}:tag:stable-5443:created_at"
17) "container_repository:{4}:tag:beta-17313:created_at"
18) "container_repository:{4}:tag:stable-4693:created_at"
redis redis/redis.socket>

18 keys are cached because the 2 other are from tags that were destroyed and we don't cache created values for tags that we know we're going to destroy.

This is working as expected 👍

🛃 Does this MR meet the acceptance criteria?

Conformity

I have included changelog trailers, or none are needed. (Does this MR need a changelog?)
[-] I have added/updated documentation, or it's not needed. (Is documentation required?)
[-] I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
[-] I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
I have self-reviewed this MR per code review guidelines.
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
I have followed the style guides.
This change is backwards compatible across updates, or this does not apply.

Availability and Testing

I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
[-] I have tested this MR in all supported browsers, or it's not needed.
[-] I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

[-] Label as security and @ mention @gitlab-com/gl-security/appsec
[-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
[-] Security reports checked/validated by a reviewer from the AppSec team

Edited Sep 03, 2021 by David Fernandez

Cleanup tags service: cache created_at values for tags that are kept