Run ClickHouse migrations during gitlab:db:configure (!138661) · Merge requests · GitLab.org / GitLab

Vladimir Shushlin requested to merge vshushlin/auto-run-ch-migrations into master Dec 04, 2023

What does this MR do and why?

This MR addresses:

Clickhouse - automatically run migrations durin... (omnibus-gitlab#8266 - closed)
Clickhouse - automatically run migrations durin... (gitlab-org/charts/gitlab#5043 - closed)

I initially tried to implement it directly on omnibus and charts: Draft: Automatically migrate ClickHouse databases (omnibus-gitlab!7272 - closed)

But then realized the both omnibus and charts right now just call the gitlab:db:configure rake task and expect it to do all the work.

So this MR just hooks into that task.

Key things to look at:

migrations moved to the db/clickhouse/migrate/main directory
we now have gitlab:clickhouse:migrate:main and gitlab:clickhouse:rollback:main rake tasks
The gitlab:clickhouse:rollback rollback task is gone, because it no longer makes sense
But there's gitlab:clickhouse:migrate which just runs migrations for all databases (only main for now)
gitlab:db:configure now also runs gitlab:clickhouse:migrate, but it skips the clickhouse migrations if the connection is not configured.
running migrations automatically is also hidden behind the feature flag, just in case.

Screenshots or screen recordings

Tests

./bin/rails gitlab:clickhouse:rollback:main VERSION=0
== 20231114142100 CreateAuditEvents: reverting ================================
== 20231114142100 CreateAuditEvents: reverted (0.0976s) =======================

== 20231106202300 ModifyCiFinishedBuildsSettings: reverting ===================
== 20231106202300 ModifyCiFinishedBuildsSettings: reverted (0.0098s) ==========

== 20230808140217 CreateCiFinishedBuildsAggregatedQueueingDelayPercentilesMv: reverting
== 20230808140217 CreateCiFinishedBuildsAggregatedQueueingDelayPercentilesMv: reverted (0.0087s)

== 20230808070520 CreateSyncCursors: reverting ================================
== 20230808070520 CreateSyncCursors: reverted (0.0949s) =======================

== 20230724064918 CreateContributionAnalyticsEventsMaterializedView: reverting
== 20230724064918 CreateContributionAnalyticsEventsMaterializedView: reverted (0.0084s)

== 20230724064832 CreateContributionAnalyticsEvents: reverting ================
== 20230724064832 CreateContributionAnalyticsEvents: reverted (0.0958s) =======

== 20230719101806 CreateCiFinishedBuildsAggregatedQueueingDelayPercentiles: reverting
== 20230719101806 CreateCiFinishedBuildsAggregatedQueueingDelayPercentiles: reverted (0.0988s)

== 20230707151359 CreateCiFinishedBuilds: reverting ===========================
== 20230707151359 CreateCiFinishedBuilds: reverted (0.1162s) ==================

== 20230705124511 CreateEvents: reverting =====================================
== 20230705124511 CreateEvents: reverted (0.1364s) ============================




mv config/click_house.yml config/click_house.yml2




./bin/rails gitlab:db:configure
Running db:migrate:main rake task
main: == [advisory_lock_connection] object_id: 182620, pg_backend_pid: 34210
main: == [advisory_lock_connection] object_id: 182620, pg_backend_pid: 34210
Running db:migrate:ci rake task
ci: == [advisory_lock_connection] object_id: 185040, pg_backend_pid: 34927
ci: == [advisory_lock_connection] object_id: 185040, pg_backend_pid: 34927
Running gitlab:clickhouse:migrate:main rake task
The 'main' ClickHouse database is not configured, skipping migrations



./bin/rails gitlab:clickhouse:migrate
Running gitlab:clickhouse:migrate:main rake task
rails aborted!
ClickHouse::Client::ConfigurationError: The database 'main' is not configured
/Users/vlad/gdk/gitlab/gems/click_house-client/lib/click_house/client.rb:95:in `block in lookup_database'
/Users/vlad/gdk/gitlab/gems/click_house-client/lib/click_house/client.rb:94:in `lookup_database'
/Users/vlad/gdk/gitlab/gems/click_house-client/lib/click_house/client.rb:100:in `instrumented_execute'
/Users/vlad/gdk/gitlab/gems/click_house-client/lib/click_house/client.rb:35:in `select'
/Users/vlad/gdk/gitlab/lib/click_house/connection.rb:11:in `select'
/Users/vlad/gdk/gitlab/lib/click_house/connection.rb:29:in `table_exists?'
/Users/vlad/gdk/gitlab/lib/click_house/migration_support/schema_migration.rb:12:in `ensure_table'
/Users/vlad/gdk/gitlab/lib/tasks/gitlab/click_house/migration.rake:73:in `migrate'
/Users/vlad/gdk/gitlab/lib/tasks/gitlab/click_house/migration.rake:16:in `block (5 levels) in <main>'
/Users/vlad/gdk/gitlab/lib/tasks/gitlab/click_house/migration.rake:35:in `block (4 levels) in <main>'
/Users/vlad/gdk/gitlab/lib/tasks/gitlab/click_house/migration.rake:33:in `each'
/Users/vlad/gdk/gitlab/lib/tasks/gitlab/click_house/migration.rake:33:in `block (3 levels) in <main>'
Tasks: TOP => gitlab:clickhouse:migrate:main
(See full trace by running task with --trace)



mv config/click_house.yml2 config/click_house.yml



./bin/rails gitlab:db:configure
Running db:migrate:main rake task
main: == [advisory_lock_connection] object_id: 182660, pg_backend_pid: 47120
main: == [advisory_lock_connection] object_id: 182660, pg_backend_pid: 47120
Running db:migrate:ci rake task
ci: == [advisory_lock_connection] object_id: 185080, pg_backend_pid: 47831
ci: == [advisory_lock_connection] object_id: 185080, pg_backend_pid: 47831
Running gitlab:clickhouse:migrate:main rake task
== 20230705124511 CreateEvents: migrating =====================================
== 20230705124511 CreateEvents: migrated (0.0203s) ============================

== 20230707151359 CreateCiFinishedBuilds: migrating ===========================
== 20230707151359 CreateCiFinishedBuilds: migrated (0.0148s) ==================

== 20230719101806 CreateCiFinishedBuildsAggregatedQueueingDelayPercentiles: migrating
== 20230719101806 CreateCiFinishedBuildsAggregatedQueueingDelayPercentiles: migrated (0.0114s)

== 20230724064832 CreateContributionAnalyticsEvents: migrating ================
== 20230724064832 CreateContributionAnalyticsEvents: migrated (0.0150s) =======

== 20230724064918 CreateContributionAnalyticsEventsMaterializedView: migrating
== 20230724064918 CreateContributionAnalyticsEventsMaterializedView: migrated (0.0172s)

== 20230808070520 CreateSyncCursors: migrating ================================
== 20230808070520 CreateSyncCursors: migrated (0.0127s) =======================

== 20230808140217 CreateCiFinishedBuildsAggregatedQueueingDelayPercentilesMv: migrating
== 20230808140217 CreateCiFinishedBuildsAggregatedQueueingDelayPercentilesMv: migrated (0.0108s)

== 20231106202300 ModifyCiFinishedBuildsSettings: migrating ===================
== 20231106202300 ModifyCiFinishedBuildsSettings: migrated (0.0342s) ==========

== 20231114142100 CreateAuditEvents: migrating ================================
== 20231114142100 CreateAuditEvents: migrated (0.0307s) =======================

== 20231129062064 CreateContributionsTable: migrating =========================
== 20231129062064 CreateContributionsTable: migrated (0.0120s) ================

== 20231129062151 CreateContributionsMv: migrating ============================
== 20231129062151 CreateContributionsMv: migrated (0.0114s) ===================

== 20231205104100 ModifyCiFinishedBuildsStartedAtDefault: migrating ===========
== 20231205104100 ModifyCiFinishedBuildsStartedAtDefault: migrated (0.0125s) ==

== 20231205104101 ModifyCiFinishedBuildsFinishedAtDefault: migrating ==========
== 20231205104101 ModifyCiFinishedBuildsFinishedAtDefault: migrated (0.0121s) =

== 20231205112200 FixInvalidCiFinishedBuildsStartedAtValues: migrating ========
== 20231205112200 FixInvalidCiFinishedBuildsStartedAtValues: migrated (0.0110s)





./bin/rails gitlab:clickhouse:migrate
Running gitlab:clickhouse:migrate:main rake task
No migrations ran.




./bin/rails gitlab:clickhouse:rollback:main STEP=2
== 20231205112200 FixInvalidCiFinishedBuildsStartedAtValues: reverting ========
== 20231205112200 FixInvalidCiFinishedBuildsStartedAtValues: reverted (0.0000s)

== 20231205104101 ModifyCiFinishedBuildsFinishedAtDefault: reverting ==========
== 20231205104101 ModifyCiFinishedBuildsFinishedAtDefault: reverted (0.0141s) =

How to set up and validate locally

In rails console enable the experiment fully

Feature.enable(:run_clickhouse_migrations_automatically)

repeat commands from the section above

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

I have evaluated the MR acceptance checklist for this MR.

Edited Dec 11, 2023 by Vladimir Shushlin

Run ClickHouse migrations during gitlab:db:configure

What does this MR do and why?

Screenshots or screen recordings

How to set up and validate locally

MR acceptance checklist

Merge request reports