Loading
Renew the ClickHouse migration lock with a heartbeat
What does this MR do and why?
The migration lock was taken once with a 1 hour TTL and never renewed. A migration process that died without releasing it blocked every retry until the TTL ran out, and a run longer than an hour silently lost the lock, letting ClickHouse workers resume mid-migration.
The lock now has a 5 minute TTL that a background thread renews every minute, for at most an hour. An orphaned lock clears within minutes, and the migrator stops before the next migration once the lock is lost.
References
ClickHouse migration lock: heartbeat test results
Tested against GDK Redis. Real timings: 5 min TTL, renewed every 1 min, renewal stops after 1h. Fast timings: 5s TTL, renewed every 1s, renewal stops after 12s.
| # | Case | Timings | Result | Lock after exit |
|---|---|---|---|---|
| 1 | Migration succeeds | real | Finished, TTL was 300s inside | Released |
| 2 | Migration raises | real | RuntimeError reached the caller |
Released |
| 3 | SIGTERM | real | SignalException, ensure ran |
Released |
| 4 | SIGKILL | real | Process died | Orphaned, TTL 300s and down to 235s after 65s (nothing renews it) |
| 5 | SIGKILL | fast | Process died | Orphaned, gone 6s later |
| 6 | Run longer than TTL (10s vs 5s) | fast | Renewed (TTL=4 at 10s), ensure_held! ok |
Released |
| 7 | Run past the 12s limit (20s) | fast | Renewal stopped, key expired, ensure_held! raised LockError |
Already expired |
| 8 | Lock taken by another UUID | fast | renew returned false, ensure_held! raised LockError |
Other holder's key left alone |
| 9 | 2 of the renew calls raise a Redis error |
fast | Kept renewing (7 calls), ensure_held! ok |
Released |
| 10 | Second migrator while the lock is held | fast | Second one got LockError |
Released |
A killed migration pod now blocks retries for at most 5 minutes instead of 1 hour (verified with test case 4).
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Adam Hegyi