Migrate Clickhouse migrations from Deploy VMs to Kubernetes
Summary
ClickHouse is a database component which is enabled in gstg and gprd. This service does not have a separate subchart or its own migration Chart. We have to run the Rake task gitlab-rake gitlab:clickhouse:migrate in an environment that has the connection information for ClickHouse.
Values required to configure ClickHouse: GitLab Helm chart documentation
Configuration for GitLab.com:
Values usage
I traced the usage of these values through the Helm chart.
The value of clickhouse.enabled is used to decide whether to include connection information in the webservice, sidekiq and toolbox chart configurations. This will cause the creation of the click_house.yml.erb file, which will be templated during the container start-up.
The boolean value clickhouse.enabled is also used to decide whether the Clickhouse specific secrets should be mounted here.
Running migrations for Clickhouse is not currently supported by the GitLab-Migrations chart. In fact, there is no script even inside CNG. Dedicated does this by running the gitlab:clickhouse:migrate Rake task on the Toolbox pod (this is the approach that they use for Rails as well):
ClickHouse is automatically configured as an initializer in Rails as part of this initializer. The only condition that is checked is the existence of the config/click_house.yml file.
ClickHouse does not have post-deployment migrations right now according to Guides: Clickhouse. However, there is support for running those post-deployment migrations in the Ansible playbooks when they are eventually introduced.
Current Setup
gstg-cny-migrations and gstg-migrations
The last migration in ClickHouse was added on 2025-07-17:
This migration was added as part of gitlab-org/gitlab!198081 (merged).
Options
The ClickHouse DB is configured for all webservice, sidekiq, and toolbox pods in both canary and the main stage.
We can use the Toolbox pod to run the rake task gitlab:clickhouse:migrate, which has all the required ClickHouse connection information already. This would be a simple change. It will require some changes to the CI configuration, similar to what we did in gitlab-com/gl-infra/k8s-workloads/gitlab-com!4581 (merged).
The better solution might be to integrate ClickHouse migrations into the GitLab-Migrations chart, and be able to configure it in the same way that we configure the Webservice, Toolbox, and Sidekiq charts.
Exit criteria
-
Update CNG code to enforce ClickHouse regular migrations before Rails start-up -
Update GitLab-Migrations Helm chart to run ClickHouse migrations -
Import chart and CNG updates into gitlab-com -
Update Deployer and run ClickHouse migrations in Kubernetes in gstg-cny and gstg -
Verify ClickHouse migrations are running correctly in Staging -
Update Deployer and run ClickHouse migrations in Kubernetes in gprd-cny and gprd