Skip to content

Step 2 of #62214: Includes logic to update namespaces storage statistics

What does this MR do?

This is the second part of #62214 (closed). This merge request includes logic to update namespaces storage statistics in an async way

Built upon https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29570

Does this MR meet the acceptance criteria?

Conformity

Performance and testing

Closes #62214 (closed)

To do (backend)

  • Whenever the statistics of a project changes, we insert a row into namespace_aggregation_schedules
    • We don't insert a new row if there's already one related to the namespace.
    • Insertion is done through a callback and with a Sidekiq job. We can't do it in the same transaction as ProjectStatistics is already involved in a large one (https://gitlab.com/gitlab-org/gitlab-ce/issues/62488)
  • After inserting the row, we schedule a Namespaces::RootStatisticsWorker job, 3 hours into the future.
  • This job will:
    • Update the namespace storage statistics through Namespaces::StatisticsRefresherService
    • Delete the related namespace_aggregation_schedules after the update
  • Create the service that will update the namespace statistics (Namespaces::StatisticsRefresherService)
  • We also need to create another Sidekiq job that will traverse any remaining rows on namespace_aggregation_schedules and schedule jobs for every pending row (Namespaces::PruneAggregationSchedulesWorker)
  • Hide changes behind a group feature flag
  • Read the interval of caching time form redis defaulting to once every 3 hours
  • Refactor and add specs.
Edited by Mayra Cabrera

Merge request reports