Meter ClickHouse consistency mutations and make the worker caps stop a run

Part of Project Bulkhead. Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636121

What does this MR do and why?

ClickHouse::EventPathsConsistencyCronWorker and ClickHouse::EventAuthorsConsistencyCronWorker repair stale paths and deleted authors in the events table by issuing ALTER TABLE ... UPDATE and ALTER TABLE ... DELETE mutations. Each mutation reads the whole table, and three of the controls around them do less than they appear to:

  • MAX_RECORD_MODIFICATIONS and MAX_AUTHOR_DELETIONS are compared only after a whole slice of 2,500 values has been collected, so a run that reaches the cap still issues the mutations for everything in the slice and only reports limit_reached.
  • The exclusive lease in ClickHouse::Concerns::ConsistencyWorker wraps only the ClickHouse iteration. The mutations and the cursor write run after it is released, so a second run can start while the first is still issuing mutations.
  • The authors worker issues its two DELETE mutations on every run, including runs with nothing to delete.

This MR adds metering unflagged and the three fixes behind a feature flag.

Metering, always on: the concern routes every mutation through execute_mutation, which increments gitlab_clickhouse_consistency_mutations_total{worker} after each ALTER TABLE returns, so a run that fails part-way through still reports the mutations it issued. Runs that issue nothing do not increment. The metric is documented in the Sidekiq metrics table.

Behind clickhouse_consistency_workers_strict_caps:

  • The paths worker checks the cap per value and stops collecting once it is reached; the authors worker trims each batch of missing ids to the remaining room under the cap, sorted so the saved cursor is the highest id kept and the rest are picked up next run.
  • perform runs process_collected_values, the cursor write and the done log inside the lease, and renews the lease before every mutation and before the cursor write, including runs that collected nothing. A lease that cannot be renewed stops the run with LeaseLostError before anything is sent, since another run may already hold it. A run that finds the lease taken logs status: :lease_taken and returns instead of raising into the Sidekiq retry.
  • The authors worker returns before building the DELETE queries when it collected nothing.

Unflagged, alongside the metering: the concern processes each ClickHouse batch in cursor order (sort_key), which for the paths worker is the trailing namespace_id of the path rather than the path string. Before, a run that stopped on the cap could save a cursor above namespaces whose paths sorted later as strings (1/1499/ before 2/20/), and the next run's namespace_id >= cursor predicate skipped them until a full pass reset the cursor. With the per-value cap that cut would have happened on every capped run, so it is fixed here rather than as a follow-up.

With the flag off, the lease, the caps and the empty DELETE behave exactly as before.

Business impact

A group transfer rewrites traversal_ids for every namespace underneath, and the next consistency run issues two ALTER TABLE ... UPDATE mutations per moved namespace on the shared events table, each a full-table rewrite: a subgroup with 30 projects is 62 mutations. The caps only reported limit_reached after the mutations were sent, the lease did not cover the mutation phase, and the authors worker issued two empty DELETEs per run.

Always on: every mutation is counted in gitlab_clickhouse_consistency_mutations_total{worker}. With clickhouse_consistency_workers_strict_caps (default off): a paths run issues at most 2 x 500 UPDATEs plus two DELETEs, an authors run deletes at most 2,000 ids, one run per worker holds the lease across mutations and cursor write and renews it before each (LeaseLostError otherwise), a second run logs status: lease_taken, and empty DELETEs are skipped. Users see nothing; the shared cluster's mutation queue stops growing with transfers.

Queries this change generates

Statements per run, captured from event_paths_consistency_cron_worker_spec.rb and event_authors_consistency_cron_worker_spec.rb (mutations_sync: 1); {name:Type} placeholders are the client's parameter binding. Redis lease commands derived from Gitlab::ExclusiveLease. The statements are unchanged; what changes is how many run and when they run relative to the lease.

SET gitlab:exclusive_lease:ClickHouse::<Worker> <uuid> NX EX 300                                          -- now wraps the whole run with the flag on
SELECT MIN(namespace_id), MAX(namespace_id) FROM event_namespace_paths
SELECT DISTINCT path FROM event_namespace_paths WHERE namespace_id >= <lower> AND namespace_id <= <upper>   -- per 100,000 ids
ALTER TABLE events UPDATE path={new_path:String} WHERE path = {old_path:String}                            -- per moved namespace, max 500 with the flag
ALTER TABLE event_namespace_paths UPDATE path={new_path:String} WHERE namespace_id = {namespace_id:UInt64}
ALTER TABLE events DELETE WHERE path IN (<paths>)
ALTER TABLE event_namespace_paths DELETE WHERE namespace_id IN ({namespace_ids:Array(UInt64)})
SELECT MIN(author_id), MAX(author_id) FROM event_authors                                                   -- authors worker
SELECT DISTINCT author_id FROM event_authors WHERE author_id >= <lower> AND author_id <= <upper>
ALTER TABLE events DELETE WHERE author_id IN ({author_ids:Array(UInt64)})                                  -- authors worker, max 2,000 ids, skipped when empty
ALTER TABLE event_authors DELETE WHERE author_id IN ({author_ids:Array(UInt64)})
EVAL <renew script> ... 300                                                                                -- flag on, before each ALTER and the cursor write
INSERT INTO sync_cursors ...                                                                               -- unchanged cursor write, inside the lease with the flag on
EVAL <cancel script> ...                                                                                   -- lease release in ensure

The PostgreSQL lookups per slice of 2,500 values (SELECT id, traversal_ids FROM namespaces WHERE id IN (<ids>), SELECT id FROM users WHERE id IN (<ids>)) are unchanged.

Threshold

The caps now bind per value: a paths run issues at most two mutations per modification up to MAX_RECORD_MODIFICATIONS (500) plus the two batched deletes, and an authors run deletes at most MAX_AUTHOR_DELETIONS (2,000) ids per run. One run per worker at a time: the lease covers the full run and is renewed before each mutation and before the cursor write, so it holds for as long as no single ClickHouse round trip exceeds the 300 s TTL; a run whose renewal fails raises before sending that mutation or the cursor rather than continuing without the lease. The iteration phase keeps its 150 s runtime budget.

Rollout

  • Rollout issue: #637955
  • The flag is gitlab_com_derisk, default off. Rollback is turning the flag off; the next run returns to the previous behaviour.
  • The metering ships unflagged so the counter is available before and during the rollout.

Verification

Run locally against ClickHouse 26.8.6.5 with mutations_sync: 1:

  • ee/spec/workers/click_house/event_paths_consistency_cron_worker_spec.rb: 21 examples, 0 failures
  • ee/spec/workers/click_house/event_authors_consistency_cron_worker_spec.rb: 21 examples, 0 failures
  • spec/workers/click_house/concerns/consistency_worker_spec.rb: 2 examples, 0 failures

The pre-existing examples run under both flag states through a shared example group, with the mutation counter asserted inside the repair examples. New examples cover the per-value cap stopping at the cap with the saved cursor and the fixed namespace asserted, a two-run repair where the path string order and the namespace_id order disagree, a second instance finding the lease taken both during process_collected_values and during the cursor write, a lease lost during collection issuing no mutation and, with nothing collected, writing no cursor, a run that finds the lease taken logging lease_taken, the authors worker issuing no mutation with nothing to delete, an over-time stop on a batch with no missing authors keeping the last scanned id as the cursor, and the flag-off counterparts of each. The CE concern spec covers the default without strict caps.

Follow-ups

  • Collapse the per-namespace events UPDATE mutations into one mutation per run (UPDATE path = transform(path, [old...], [new...], path) WHERE path IN (old...)), which turns 2N mutations into N+1.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Chandra Saripaka

Merge request reports

Loading
Loading