Refactor NamespaceStatistics into CE

Vijay Hawoldarrequested to merge
vij-refactor-namespace-statistics-ce into master

What does this MR do and why?

Context

As part of https://gitlab.com/groups/gitlab-org/-/epics/7011 we are going to begin displaying storage usage for certain containers that were previously unexposed, starting with Dependency Proxy in https://gitlab.com/gitlab-org/gitlab/-/issues/348172

To achieve this, we're re-using the existing storage calculations at the Namespace level via NamespaceStatistics and calculating the total across all namespaces in a hierarchy via Namespace::RootStorageStatistics

The existing NamespaceStatistics implementation is within the EE namespace because at the time of creation, the only namespace (group) level statistic being tracked was group wiki size, which is a Premium+ feature.

Dependency Proxy is available for groups on all tiers.

This MR

So that we can calculate and store dependency proxy totals at the namespace and root namespace statistic levels for CE, this MR makes the following changes:

  • move most of the NamespaceStatistic implementation into a new CE class of the same name,
  • convert the old NamespaceStatistic class into an EE module to cater for the group wiki aspect
  • add dependency_proxy_size calculations to the NamespaceStatistic refresh
  • move the NamespaceStatistic calculations from the EE RootStorageStatistic to the CE version
  • add NamespaceStatistic#dependency_proxy_size to the RootStorageStatistic calculations

I intend to handle the update of NamespaceStatistics when dependency proxy assets are added/removed in a separate MR.

How to set up and validate locally

  1. Create some dependency proxy records with a size in the DB: 
    dpm = DependencyProxy::Manifest.new(size: 100, group: my_group)
dpb = DependencyProxy::Blob.new(size: 200, group: my_group)

dpm.save(validate: false) # bypasses the need to satisfy DP:M validations
dpb.save(validate: false) # bypasses the need to satisfy DP:B validations
  2. Check storage sizes and trigger a refresh 
    namespace_statistics = NamespaceStatistics.find_or_create_by!(namespace_id: group.id)
namespace_statistics.storage_size
namespace_statistics.dependency_proxy_size

namespace_statistics.refresh!

namespace_statistics.storage_size # expect 300
namespace_statistics.dependency_proxy_size # expect 300
  3. Check root storage statistics and trigger a refresh 
    root_statistics = Namespace::RootStorageStatistics.find_or_create_by!(namespace_id: group.id)
root_statistics.dependency_proxy_size

root_statistics.recalculate!

root_statistics.dependency_proxy_size # expect 300

If you have a group on a premium or ultimate plan, you could optionally create a group wiki and re-run the checks above but also check wiki_size. It's worth noting that at the RootStorageStatistics#wiki_size will also include project level wikis.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vijay Hawoldar

Merge request reports