GitLab Next

What does this MR do?

This is the second part of #62214 (closed). This merge request includes logic to update namespaces storage statistics in an async way

Built upon https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29570

Does this MR meet the acceptance criteria?

Conformity

• [-] Changelog entry
• [-] Documentation created/updated or follow-up review issue created
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Performance and testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• [-] Tested in all supported browsers

Closes #62214 (closed)

To do (backend)

• Whenever the statistics of a project changes, we insert a row into namespace_aggregation_schedules
  • We don't insert a new row if there's already one related to the namespace.
  • Insertion is done through a callback and with a Sidekiq job. We can't do it in the same transaction as ProjectStatistics is already involved in a large one (https://gitlab.com/gitlab-org/gitlab-ce/issues/62488)
• After inserting the row, we schedule a Namespaces::RootStatisticsWorker job, 3 hours into the future.
• This job will:
  • Update the namespace storage statistics through Namespaces::StatisticsRefresherService
  • Delete the related namespace_aggregation_schedules after the update
• Create the service that will update the namespace statistics (Namespaces::StatisticsRefresherService)
• We also need to create another Sidekiq job that will traverse any remaining rows on namespace_aggregation_schedules and schedule jobs for every pending row (Namespaces::PruneAggregationSchedulesWorker)
• Hide changes behind a group feature flag
• Read the interval of caching time form redis defaulting to once every 3 hours
• Refactor and add specs.