Add runner usage breakdown to dashboard
What does this MR do and why?
This fix ensures the right elements are shown when ClickHouse is configured for the runners dashboard as well as making the dashboard layout when they are not.
When Clickhouse analytics is enabled, the following dashboard elements become available:
- Runner Usage CSV export
- Wait time to pick a job history chart
Changelog: fixed
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Runner Usage Panel |
---|
Full dashboard with Clickhouse enabled | Full dashboard with Clickhouse disabled |
---|---|
How to set up and validate locally
The dashboard compiles data from our usual Postgres database and the new Clickhouse integration. Since not all of our self-hosted users will have Clickhouse enabled, we offer the two variants of this dashboard.
First, ensure your instance has a GitLab Ultimate license.
1. Generating seed data
First, we want a good amount of recent, long duration jobs so they show up in our data.
- Run
bin/rake "gitlab:seed:runner_fleet"
(more about this command)
We can test two states of the dashboard, with ClickHouse enabled and disabled.
2. Test without Clickhouse (optional)
By default, your instance may not have Clickhouse configured.
-
Ensure you don't have a file at
config/click_house.yml
. -
Restart GDK if needed
-
Visit the dashboard (e.g. http://gdk.test:3000/admin/runners/dashboard)
-
Confirm the "Runner Usage" panel is not visible, and it is replaced with the "Most recent failures" panel.
3. Test with Clickhouse
- Locally, create a file called
config/click_house.yml
, with the following contents:
test:
main:
database: gitlab_clickhouse_test
url: 'http://localhost:8123'
username: default
development:
main:
database: gitlab_clickhouse_development
url: 'http://localhost:8123'
username: default
This will tell your instance to look for a clickhouse database at http://localhost:8123
. Even if you don't have one, the dashboard will show in the desired state (with an error at the top).
-
Restart your instance (e.g.
gdk restart
) -
Run the Clickhouse migrations
bundle exec rake gitlab:clickhouse:migrate
-
Force the syncronization service so the Clickhouse data is up to date, in the Rails console (
rails c
), type:
::Ci::Build.where.not(finished_at: nil).find_each{|build| ::Ci::FinishedBuildChSyncEvent.upsert({ build_id: build.id, build_finished_at: build.finished_at }, unique_by: [:build_id, :partition]) }
ClickHouse::DataIngestion::CiFinishedBuildsSyncService.new.execute
- Visit the dashboard (e.g. http://gdk.test:3000/admin/runners/dashboard)
Related to #421457 (closed)