Skip to content

Implement value stream dashbourd count service

Adam Hegyi requested to merge 402739-implement-dashboard-count-worker into master

What does this MR do and why?

Note: not user facing change. The worker will be enabled in the next MR.

This MR implements the count collection worker for value stream dashboard. The worker will iterate over the selected groups and records counts of various items in the group hierarchy. This MR contains two metrics:

  • Number of projects per group
  • Number of issues per project namespace

To avoid timeouts and long running sidekiq jobs the service that implements the counting logic uses batch processing which is stopped when the time limit is reached. When this happens, the current "state" will be persisted (not implemented yet) and the processing will retry later.

How to set up and validate locally

Execute the service for a group:

group = Group.find(9970)
aggregation = Analytics::ValueStreamDashboard::Aggregation.find_or_create_by(namespace_id: group.id)

cursor = {}

runtime_limiter = Analytics::CycleAnalytics::RuntimeLimiter.new(30.seconds)
loop do

  service = Analytics::ValueStreamDashboard::TopLevelGroupCounterService.new(
    aggregation: aggregation,
    runtime_limiter: runtime_limiter,
    raw_cursor: cursor
  )

  service_response = service.execute

  if service_response.payload[:result] == :interrupted
    runtime_limiter = Analytics::CycleAnalytics::RuntimeLimiter.new(30.seconds)
    cursor = service_response.payload[:cursor]
    Rails.logger.info "over time - restart the runtime limiter"
    Rails.logger.info "cursor: #{cursor}"
  else
    Rails.logger.info "count completed"
    Rails.logger.info "cursor: #{cursor}"
    Rails.logger.info "cursor: #{service_response.payload}"
    break
  end
end

Compare the collected counts with the actual counts, they should match:

# fresh issue count
puts Issue.where(project_id: group.all_projects.select(:id)).count
# aggregated count
puts Analytics::ValueStreamDashboard::Count.count_for(:issues)

# fresh project count
puts group.all_projects.count
# aggregated count
Analytics::ValueStreamDashboard::Count.count_for(:projects)

Database

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #402739 (closed)

Edited by Adam Hegyi

Merge request reports