Cache container repositories size on namespace
What does this MR do and why?
This MR adds caching to the Namespace#container_repositories_size
method, which is invalidated when we receive an appropriate (push or delete) notification to the Container Registry Events API, for any project within it's hierarchy.
This change is being added because otherwise RootStorageStatistics
will unnecessarily call the Container Registry API whenever there are unrelated statistic changes, which is a heavy computation and will result in many additional calls to the API that we can simply avoid.
Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/361960#note_959932406.
Additional context
Whenever a ProjectStatistics::NAMESPACE_RELATABLE_COLUMN is updated (e.g. when the repository size changes), the RootStorageStatistics
record for the project's root namespace, will have it's statistics recalculated.
Amongst other things, the RootStorageStatistics#recalculate
method will query the Container Registry API to find out the deduplicated size for the entire namespace hierarchy and store it in container_registry_size
column.
sum
the project_statistics#container_registry_size
values?
Why don't we Due to the way in which the Container Registry shares images between projects/groups, this would not result in an accurate storage size for an entire namespace hierarchy.
See &7578 (comment 870274096) for more detail/explanations.
How to set up and validate locally
- Firstly, either follow the Container Registry setup steps described in !83980 (merged) so you can have a full end-to-end setup
or, the quicker/easier method: modify Namespace#container_repositories_size
so that it doesn't call the registry API and bypasses the various other checks which are untouched in this MR:
def container_repositories_size
strong_memoize(:container_repositories_size) do
Rails.cache.fetch(container_repositories_size_cache_key, expires_in: 1.day) do
1234
end
end
end
- Enable local GDK caching:
bin/rails dev:cache
- Call the method to check it still works:
> Namespace.last.container_repositories_size
=> 1234
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.