feat(clickhouse): one replicated switch for self-managed HA clusters

What does this MR do and why?

On a self-managed ClickHouse cluster the graph tables were created with plain engines, so rows stayed on the replica that took the write, and quorum reads or writes failed on transient cluster errors. This MR replaces the quorum_writes flag with one replicated switch per connection. With it on, the graph gets replicated table engines, both connections write with quorum and read with sequential consistency, and the client retries the transient quorum and Keeper errors. Off by default, so nothing changes for ClickHouse Cloud or a single node.

Related Issues

Relates to #1084 (closed). Chart side: orbit-helm-charts!160 (merged) (clickhouse.ha.enabled: true).

Testing

Unit tests cover the engine rewrite, the retry classification, and the retry loop. Validated live on a 3-replica ClickHouse 26.7 cluster with embedded Keeper behind haproxy on the dev VM, with the whole GDK stack (Rails migrations, Siphon, GitLab Orbit) pointed at it:

Check	Result
Graph tables on every replica (rerun on the final commit, fresh Replicated database)	42 of 42 ReplicatedReplacingMergeTree, default ZooKeeper path
Rows on the 3 replicas after indexing	identical: 184 projects, 756,021 definitions, 3,521,868 code edges
Orbit REST battery through the load balancer	12 of 12 pass
Quorum collisions absorbed by the client retry in 30 min of indexing	about 630 per replica (error 286), 2 to 6 (error 289), no write failed
Kill one replica under live indexing, 40 API queries over 40 s	10 failed before the Keeper retry, 0 after
Killed replica rejoins and catches up	8 s and 13 s in two drills
Kill a replica 7 tables into a fresh schema migration (final commit)	migration completes, 42 of 42 replicated tables and version active on all 3 replicas

Performance Analysis

No effect while the switch is off, which is every current deployment. With it on, the costs are the ones of quorum writes: more parts, serialized inserts per table, and Arrow batches held until the insert is acknowledged so a retry can resend them.

• This merge request does not introduce any performance regression. If a performance regression is expected, explain why.

Agent context

One switch instead of two

quorum_writes (from !2153 (merged)) only set session settings. A Replicated database replicates DDL and nothing else, so the data still needed replicated table engines. There is no case for one without the other, so replicated does both. The rename is a config break, but no deployment sets quorum_writes today. ClickHouse has no server-side conversion: database_replicated_allow_only_replicated_engine only rejects plain engines (see InterpreterCreateQuery.cpp), so the rewrite is ours.

Where the engines are rewritten

In the schema builder, not in the client. GraphSchema::from_ontology_replicated maps every Engine (tables, materialized views, unversioned tables) through Engine::replicated, and the version table renders its engine the same way. The client only carries a typed is_replicated() flag from its connection config, which the three DDL entry points read; no SQL is parsed. The fingerprints, the xtask DDL dump, and the query path keep the plain engines.

Where the retry lives

The retry sits in the client on execute, fetch_arrow_with_summary, the two streamed fetches, and the streaming insert. Every DDL, checkpoint read, schema-version read, and graph write goes through one of those, so nothing in the indexer or webserver changed. The streamed fetches prefetch the first chunk: a quorum error arrives with the first response, before any rows. In quorum mode the streaming insert clones the Arrow batches per attempt, which is a refcount bump. Off quorum mode the batches are moved as before.

What a replica loss looks like to the client

Killing a replica during a fresh migration on the dev cluster surfaced four error shapes, one per attempt, until the migration went through: a dropped connection (Network), QUERY_WAS_CANCELLED from the replica that was shutting down, the load balancer's 503 Service Unavailable body, and the 180 s distributed_ddl_task_timeout wait ending in "DDL task ... is not finished on 1 of 3 hosts". Replicated mode retries all of them plus the breaker's transient set. Every DDL is IF NOT EXISTS/IF EXISTS except the partition attach, which opts out.

Keeper errors

Killing the replica that hosted the Keeper leader expired every open coordination session for a few seconds. Reads with select_sequential_consistency need Keeper, so 10 of 40 API queries failed with code 999 during that window. Session expired, Connection loss, and Operation timeout now retry like 286 and 289. The second drill on the rebuilt binary returned 40 of 40.

Retry and idempotency

Codes 286 and 289 are raised before any write. The Keeper texts can fire after blocks were committed, so a retried INSERT ... SELECT (the checkpoint seed during a clone migration) can insert the same rows twice. The target is a ReplacingMergeTree, so FINAL collapses them. Arrow inserts resend identical bytes and ClickHouse deduplicates the block. Version writes carry a deduplication token.

Dead code

fetch_arrow_stream and query_arrow_stream had no callers and are gone. The whole circuit-breaking wrapper (CircuitBreakingClient) has no callers outside its crate either; left for a follow-up.

ClickHouse 26.7 note

Replaying the GitLab Rails ClickHouse migrations on 26.7 needs allow_dimensions_outside_sorting_key = 1 in the server merge_tree settings, or the AggregatingMergeTree migrations fail with code 36. Documented in the self-managed HA section. The minmax statistics rejection from July is gone in 26.7.2.