Skip to content

Cache container repositories size on namespace

Vijay Hawoldar requested to merge vij-cache-container-repositories-size into master

What does this MR do and why?

This MR adds caching to the Namespace#container_repositories_size method, which is invalidated when we receive an appropriate (push or delete) notification to the Container Registry Events API, for any project within it's hierarchy.

This change is being added because otherwise RootStorageStatistics will unnecessarily call the Container Registry API whenever there are unrelated statistic changes, which is a heavy computation and will result in many additional calls to the API that we can simply avoid.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/361960#note_959932406.

Additional context

Whenever a ProjectStatistics::NAMESPACE_RELATABLE_COLUMN is updated (e.g. when the repository size changes), the RootStorageStatistics record for the project's root namespace, will have it's statistics recalculated.

Amongst other things, the RootStorageStatistics#recalculate method will query the Container Registry API to find out the deduplicated size for the entire namespace hierarchy and store it in container_registry_size column.

Why don't we sum the project_statistics#container_registry_size values?

Due to the way in which the Container Registry shares images between projects/groups, this would not result in an accurate storage size for an entire namespace hierarchy.

See &7578 (comment 870274096) for more detail/explanations.

How to set up and validate locally

  1. Firstly, either follow the Container Registry setup steps described in !83980 (merged) so you can have a full end-to-end setup

or, the quicker/easier method: modify Namespace#container_repositories_size so that it doesn't call the registry API and bypasses the various other checks which are untouched in this MR:

  def container_repositories_size
    strong_memoize(:container_repositories_size) do
      Rails.cache.fetch(container_repositories_size_cache_key, expires_in: 1.day) do
        1234
      end
    end
  end
  1. Enable local GDK caching:
bin/rails dev:cache
  1. Call the method to check it still works:
>  Namespace.last.container_repositories_size
=> 1234

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vijay Hawoldar

Merge request reports