VSD - PoC aggregate counts for overview widget

Problem

We currently don't have a performant way to provide high level counts for group metrics i.e. Users, Projects, Groups, Issues, MRs and Pipelines

Solution

PoC aggregating these metics on a project level, so the values can be efficiently queried on the group level.

How

Use batching strategies to iteratively count records for a project: https://docs.gitlab.com/ee/development/database/iterating_tables_in_batches.html

The batching logic might be stopped and resumed at any point in case we run over time (we have a 5 minute limit / job due to error budgets).

On the high-level we should have something like this (issue count):

top_level_group = Group.find(cursor[:group_id])
top_level_group.all_projects.where(project_id >= cursor[:project_id]).each do |project|
  project_count = 0
  project.issues.where(id >= cursor[:issue_id]).each_batch do |issues|
    project_count += issues.count

    if overtime?
      persist_cursor
      return
    end
  end

  # persist the count
end

When the 5 minute runtime is up, we need to store the current top_level_group, project and latest issues.id (let's call these the cursor) so at a later step, we can continue the processing.

Expected output: scripts or merged utility methods implementing the generic counting functionality.

Open question

• Can we aggregate data only for licensed groups?