Skip to content

Introduce worker to import finished builds to ClickHouse

Pedro Pombeiro (OOO from July 16th till Aug 7th) requested to merge pedropombeiro/421201/implement-ch-worker into master
Sequence MR
1 Create p_ci_finished_build_ch_sync_events table (!131902 - merged)
2 Add service to sync builds to ClickHouse (!132010 - merged)
3 ◀ you are here Introduce worker to import finished builds to C... (!132436 - merged)
4 omnibus-gitlab!7169 (merged)
5 gitlab-org/charts/gitlab!3414 (closed)

What does this MR do and why?

This MR introduces a worker that leverages the service being introduced in !131902 (merged) to import finished builds to ClickHouse. This worker is only meant for GitLab.com for now (since customers won't have a ClickHouse installation), and is a GitLab Ultimate feature.

The worker is not marked as CPU-intensive, due to the ratio being below 30%:

image

EE: true

Closes Cron worker to ingest data from ci_builds to Cl... (#421201 - closed)

Screenshots

Sidekiq job executing:

image.png

image

image

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

NOTE: These steps were already tested in the upstream MR !132010 (merged), the only thing that this MR adds is the worker that will call the service

  1. Go to the shell in your GDK gitlab directory and run bundle exec rake "gitlab:seed:runner_fleet". This will seed your GDK with some runners and jobs required for testing this MR.

  2. Enabled the FF:

    Feature.enable(:ci_data_ingestion_to_click_house)

  3. Create a new Ci::FinishedBuildChSyncEvent record for each finished build on the GDK console:

    Ci::Build.include EachBatch

Ci::Build.finished.where.not(finished_at: nil).order(finished_at: :asc).each_batch(of: 5000) { |batch| batch = batch.map { |build| Ci::FinishedBuildChSyncEvent.new(build_id: build.id, build_finished_at: build.finished_at) }; Ci::FinishedBuildChSyncEvent.transaction { batch.each(&:save) } }

  4. Invoke the service to import all the finished builds:

    ClickHouse::CiFinishedBuildsSyncWorker.new.perform

  5. The worker should import all the finished builds into the ClickHouse ci_finished_builds table.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Pedro Pombeiro (OOO from July 16th till Aug 7th)

Merge request reports