VSD - PoC aggregate counts for overview widget
Problem
We currently don't have a performant way to provide high level counts for group metrics i.e. Users
, Projects
, Groups
, Issues
, MRs
and Pipelines
Solution
PoC aggregating these metics on a project level, so the values can be efficiently queried on the group level.
How
Use batching strategies to iteratively count records for a project: https://docs.gitlab.com/ee/development/database/iterating_tables_in_batches.html
The batching logic might be stopped and resumed at any point in case we run over time (we have a 5 minute limit / job due to error budgets).
On the high-level we should have something like this (issue count):
top_level_group = Group.find(cursor[:group_id])
top_level_group.all_projects.where(project_id >= cursor[:project_id]).each do |project|
project_count = 0
project.issues.where(id >= cursor[:issue_id]).each_batch do |issues|
project_count += issues.count
if overtime?
persist_cursor
return
end
end
# persist the count
end
When the 5 minute runtime is up, we need to store the current top_level_group
, project
and latest issues.id
(let's call these the cursor) so at a later step, we can continue the processing.
Expected output: scripts or merged utility methods implementing the generic counting functionality.
Open question
-
Can we aggregate data only for licensed groups?