Step 2 of #62214: Includes logic to update namespaces storage statistics
What does this MR do?
This is the second part of #62214 (closed). This merge request includes logic to update namespaces storage statistics in an async way
Built upon https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29570
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry
- [-] Documentation created/updated or follow-up review issue created
- Code review guidelines
- Merge request performance guidelines
- Style guides
- Database guides
- Separation of EE specific content
Performance and testing
- Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
- [-] Tested in all supported browsers
Closes #62214 (closed)
backend)
To do (-
Whenever the statistics of a project changes, we insert a row into
namespace_aggregation_schedules
- We don't insert a new row if there's already one related to the namespace.
- Insertion is done through a callback and with a Sidekiq job. We can't do it in the same transaction as
ProjectStatistics
is already involved in a large one (https://gitlab.com/gitlab-org/gitlab-ce/issues/62488)
-
After inserting the row, we schedule a
Namespaces::RootStatisticsWorker
job,3
hours into the future. -
This job will:
- Update the namespace storage statistics through
Namespaces::StatisticsRefresherService
- Delete the related
namespace_aggregation_schedules
after the update
- Update the namespace storage statistics through
-
Create the service that will update the namespace statistics (
Namespaces::StatisticsRefresherService
) -
We also need to create another Sidekiq job that will traverse any remaining rows on
namespace_aggregation_schedules
and schedule jobs for every pending row (Namespaces::PruneAggregationSchedulesWorker
) - Hide changes behind a group feature flag
- Read the interval of caching time form redis defaulting to once every 3 hours
- Refactor and add specs.