Step 2 of #62214: Includes logic to update namespaces storage statistics
What does this MR do?
This is the second part of #62214 (closed). This merge request includes logic to update namespaces storage statistics in an async way
Built upon https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29570
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry
- [-] Documentation created/updated or follow-up review issue created
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Performance and testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
Closes #62214 (closed)
backend)
To do (-
Whenever the statistics of a project changes, we insert a row into namespace_aggregation_schedules
- We don't insert a new row if there's already one related to the namespace.
- Insertion is done through a callback and with a Sidekiq job. We can't do it in the same transaction as
ProjectStatistics
is already involved in a large one (https://gitlab.com/gitlab-org/gitlab-ce/issues/62488)
-
After inserting the row, we schedule a Namespaces::RootStatisticsWorker
job,3
hours into the future. -
This job will: - Update the namespace storage statistics through
Namespaces::StatisticsRefresherService
- Delete the related
namespace_aggregation_schedules
after the update
- Update the namespace storage statistics through
-
Create the service that will update the namespace statistics ( Namespaces::StatisticsRefresherService
) -
We also need to create another Sidekiq job that will traverse any remaining rows on namespace_aggregation_schedules
and schedule jobs for every pending row (Namespaces::PruneAggregationSchedulesWorker
) -
Hide changes behind a group feature flag -
Read the interval of caching time form redis defaulting to once every 3 hours -
Refactor and add specs.
Edited by Mayra Cabrera