Skip to content

Add service for calculating most used runners

Vladimir Shushlin requested to merge vshushlin/top-runners-service into master

What does this MR do and why?

Add service calculating most used runner in the date range. Will be used in API for "most used runners" (#437659 - closed)

Very similar to GetUsageByProjectService, but I decided to reimplement it, and not to try extracting the common parts:

  • it has its own MATERIALIZED VIEW (we may explore using projections on the same view later)
  • the service is simpier because it doesn't allow to provide group_by_fields and some filters

The actual API will be implemented in the next MR similar to Add Graphql API for runner usage by project (!142333 - merged)

Migrations log

./bin/rails gitlab:clickhouse:rollback:main STEP=2
== 20240122131202 CreateMostUsedRunnersDailyMv: reverting =====================
== 20240122131202 CreateMostUsedRunnersDailyMv: reverted (0.0162s) ============


== 20240122131201 CreateMostUsedRunnersDaily: reverting =======================
== 20240122131201 CreateMostUsedRunnersDaily: reverted (0.0971s) ==============

./bin/rails gitlab:clickhouse:migrate
Running gitlab:clickhouse:migrate:main rake task
== 20240122131201 CreateMostUsedRunnersDaily: migrating =======================
== 20240122131201 CreateMostUsedRunnersDaily: migrated (0.0307s) ==============

== 20240122131202 CreateMostUsedRunnersDailyMv: migrating =====================
== 20240122131202 CreateMostUsedRunnersDailyMv: migrated (0.0125s) ============

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

I was relying on tests, but if you want to play with it:

  1. Follow https://docs.gitlab.com/ee/development/database/clickhouse/clickhouse_within_gitlab.html#gdk-setup up to and including running migrations
  2. Run the following in the rails console:
::Ci::Build.where.not(finished_at: nil).find_each{|build| ::Ci::FinishedBuildChSyncEvent.upsert({ build_id: build.id, build_finished_at: build.finished_at }, unique_by: [:build_id, :partition]) }
ClickHouse::DataIngestion::CiFinishedBuildsSyncService.new.execute
  1. Run the service itself.
Edited by Vladimir Shushlin

Merge request reports