Clickhouse - use exclusive lease for migrations and data ingestion
In Clickhouse - basic migrations support (#428124 - closed) we introduced the basic migrations support
We should also ensure that:
- only one process can execute the migrations in a given time
- Ideally, data ingestion workers are paused during the migration execution. Before starting migration:
- Pause ClickHouse workers based on presence of
ClickHouse::MigrationSupport::Migrator::LEASE_KEYin Redis. - Create
RegisterClickHouseWorkermiddleware that registers ClickHouse workers in aclick_house:workers:active_workersRedis list with an appropriate TTL score while the worker is active. - Create
ClickHouseWorkerconcern that enablesClickHouseRegisterWorker. - Ensure that Redis list is empty before starting migration, i.e.
ZRANGEBYSCORE click_house:workers:active_workers {now} +infreturns empty.
- Pause ClickHouse workers based on presence of
We can try to achieve this by using already existing mechanism of exclusive locks
Edited by Pedro Pombeiro