Discussion: move CI Analytics database architecture to CE
We currently have 2 features built on top of the ClickHouse CI Analytics database:
- Runner fleet dashboard
- Shared Runners compute minutes export
Both of them require EE and the Ultimate license. All the data ingestion works only on EE instances. If you run CE, your ClickHouse database will be empty.
But we plan to build more features on top of it, e.g. project CI Analytics page.
Proposal
If we move data ingestion to the CE:
- once you upgrade to EE, you can benefit from the historical data
- we can use ClickHouse for CE features
Cons:
- CE instances will do more work even if they never use our Analytics features. There's slight overhead with creating and storing temporary ingestion data. We do recycle that data though.
Required code changes
Move the following to CE:
-
::Ci::FinishedBuildChSyncEvent
creation https://gitlab.com/gitlab-org/gitlab/blob/c77a90984a8d7795b5260cc0d911d52246dec4e6/ee/app/workers/ee/ci/build_finished_worker.rb#L24 -
CiFinishedBuildsSyncService
https://gitlab.com/gitlab-org/gitlab/blob/c2cc77de61977853104ddc507968bb2e06019ed0/ee/app/services/click_house/data_ingestion/ci_finished_builds_sync_service.rb#L5 -
CiFinishedBuildsSyncCronWorker
https://gitlab.com/gitlab-org/gitlab/blob/380bc649eea86db3f013b456edd925e9cecf8b88/ee/app/workers/click_house/ci_finished_builds_sync_cron_worker.rb#L4 -
CiFinishedBuildsSyncWorker
https://gitlab.com/gitlab-org/gitlab/blob/8df7708cb599bced24cb2d3cbccf93e91438b3a0/ee/app/workers/click_house/ci_finished_builds_sync_worker.rb#L4
Run the worker in CE:
-
Settings.cron_jobs['click_house_ci_finished_builds_sync_worker']['job_class'] = 'ClickHouse::CiFinishedBuildsSyncCronWorker'
https://gitlab.com/gitlab-org/gitlab/blob/853af39e669370e90c86b30857fbc3e0aa144acf/config/initializers/1_settings.rb#L885 (move this line up out ofGitlab.ee?