Skip to content

Clickhouse - use exclusive lease for migrations and data ingestion

In Clickhouse - basic migrations support (#428124 - closed) we introduced the basic migrations support

We should also ensure that:

  • only one process can execute the migrations in a given time
  • Ideally, data ingestion workers are paused during the migration execution. Before starting migration:
    • Pause ClickHouse workers based on presence of ClickHouse::MigrationSupport::Migrator::LEASE_KEY in Redis.
    • Create RegisterClickHouseWorker middleware that registers ClickHouse workers in a click_house:workers:active_workers Redis list with an appropriate TTL score while the worker is active.
    • Create ClickHouseWorker concern that enables ClickHouseRegisterWorker.
    • Ensure that Redis list is empty before starting migration, i.e. ZRANGEBYSCORE click_house:workers:active_workers {now} +inf returns empty.

We can try to achieve this by using already existing mechanism of exclusive locks

Edited by Pedro Pombeiro