Skip to content

Add runner usage breakdown to dashboard

What does this MR do and why?

This fix ensures the right elements are shown when ClickHouse is configured for the runners dashboard as well as making the dashboard layout when they are not.

When Clickhouse analytics is enabled, the following dashboard elements become available:

  • Runner Usage CSV export
  • Wait time to pick a job history chart

Changelog: fixed

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Runner Usage Panel
Full dashboard with Clickhouse enabled Full dashboard with Clickhouse disabled
image image
image image

How to set up and validate locally

The dashboard compiles data from our usual Postgres database and the new Clickhouse integration. Since not all of our self-hosted users will have Clickhouse enabled, we offer the two variants of this dashboard.

First, ensure your instance has a GitLab Ultimate license.

1. Generating seed data

First, we want a good amount of recent, long duration jobs so they show up in our data.

  1. Run bin/rake "gitlab:seed:runner_fleet" (more about this command)

We can test two states of the dashboard, with ClickHouse enabled and disabled.

2. Test without Clickhouse (optional)

By default, your instance may not have Clickhouse configured.

  1. Ensure you don't have a file at config/click_house.yml.

  2. Restart GDK if needed

  3. Visit the dashboard (e.g. http://gdk.test:3000/admin/runners/dashboard)

  4. Confirm the "Runner Usage" panel is not visible, and it is replaced with the "Most recent failures" panel.

3. Test with Clickhouse

  1. Locally, create a file called config/click_house.yml, with the following contents:
test:
  main:
    database: gitlab_clickhouse_test
    url: 'http://localhost:8123'
    username: default
development:
  main:
    database: gitlab_clickhouse_development
    url: 'http://localhost:8123'
    username: default

This will tell your instance to look for a clickhouse database at http://localhost:8123. Even if you don't have one, the dashboard will show in the desired state (with an error at the top).

  1. Restart your instance (e.g. gdk restart)

  2. Run the Clickhouse migrations bundle exec rake gitlab:clickhouse:migrate

  3. Force the syncronization service so the Clickhouse data is up to date, in the Rails console (rails c), type:

::Ci::Build.where.not(finished_at: nil).find_each{|build| ::Ci::FinishedBuildChSyncEvent.upsert({ build_id: build.id, build_finished_at: build.finished_at }, unique_by: [:build_id, :partition]) }
ClickHouse::DataIngestion::CiFinishedBuildsSyncService.new.execute
  1. Visit the dashboard (e.g. http://gdk.test:3000/admin/runners/dashboard)

Related to #421457 (closed)

Edited by Miguel Rincon

Merge request reports