feat(clickhouse): add quorum_writes for self-managed replicated clusters

What does this MR do and why?

On a self-managed ClickHouse cluster with more than one replica, INSERT FROM SELECT statement could not run reliably with quorum writes. An insert may not have hit all nodes which would leave a gap in the database.

Related Issues

Relates to #1084 (closed)

Testing

Unit tests cover the emitted insert SQL with the switch on and off, the config expansion, and the precedence rule that an explicit session setting still wins. Workspace clippy, cargo fmt --check, and mise test:fast (2361 tests) pass locally.

Performance Analysis

No effect while the switch is off, which is every current deployment including production. Turning it on trades throughput for correctness: async batching is gone, so writes create more parts and more merge work, and quorum inserts are serialized. The runbook documents both costs.

• This merge request does not introduce any performance regression. If a performance regression is expected, explain why.

Agent context — long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered

Origin

siphon#254 validated Siphon against a 3-replica ReplicatedMergeTree cluster and landed on four settings: insert_quorum=auto, insert_quorum_parallel=0, select_sequential_consistency=1, async_insert=0. Auditing GKG against that list found three of the four already reachable through session_settings, and the fourth unreachable.

Why async_insert=0 was unreachable

session_settings lands on the base client and the clickhouse crate serializes it into URL query parameters (clickhouse-0.15.1/src/query.rs:234-236), so it reaches every statement. But a SETTINGS clause in the query text outranks a URL parameter. Verified against a local server (ClickHouse 26.3.9.8):

curl -s "http://127.0.0.1:8123/?async_insert=0" \
  --data-binary "SELECT getSetting('async_insert') SETTINGS async_insert=1"
→ true

Three write paths put async_insert=1 into that clause or into a per-query override, all of which beat config:

Path	Mechanism
Bulk Arrow writes, crates/indexer/src/clickhouse/writer.rs:127	SQL SETTINGS from insert_overrides; overrides beat config in insert_settings_clause
Code stale-data cleaner, crates/indexer/src/modules/code/stale_data_cleaner.rs:195	same override path
SDLC checkpoint writes, crates/indexer/src/checkpoint.rs:152	per-query with_setting, which overwrites the same-keyed session value

Paths that were already fine and are untouched: insert_query() callers (code checkpoints, namespace deletion), query().execute() statements such as the stale-edge tombstone sweep and migration DDL, and all reads.

Detection reads the settings, not a flag

has_quorum_insert_setting derives the mode from the effective session and insert settings rather than taking a boolean down the call chain. That closes the footgun where an operator hand-sets the quorum trio in session_settings without the new switch: async inserts are suppressed either way. insert_quorum=0 is ClickHouse's "no quorum" value and does not trigger it.

Graph-only

The field is documented for the graph connection. GKG only writes to its own graph tables; the datalake is written by Siphon, which applies its own quorum settings. It still lives on the shared ClickHouseConfiguration, so datalake.quorum_writes exists in the generated schema. Making it structurally graph-only needs a separate config type, which felt out of scope here. If datalake reads should fail loudly on a lagging replica instead of returning stale Siphon rows, select_sequential_consistency in datalake.session_settings already works with no code change.

Naming

An earlier draft named the flag after the effect (async_insert_supported). Naming it after the deployment fact reads better end to end: configuration, client, and call sites all talk about quorum writes, and dropping the async settings is named only at the point where it happens.

Files

• crates/gkg-server-config/src/clickhouse.rs — the new field and its operator-facing documentation, which flows into the generated config schema.
• crates/clickhouse-client/src/configuration.rs — expands the flag into the three session settings, with explicit session_settings overlaid on top.
• crates/clickhouse-client/src/arrow_client.rs — detection plus suppression in the insert SQL clause and in insert_query().
• crates/indexer/src/checkpoint.rs — skips its async pins under quorum writes.
• docs/dev/runbooks/server_configuration.md — when to set it, what it applies, what it costs.

Not covered

Three further self-managed gaps from the same audit, all better handled under #1084 (closed) than bundled here: no retry on error 289 (REPLICA_IS_NOT_IN_QUORUM), the Cloud cluster name hardcoded for ON CLUSTER in migration_completion.rs:50, and insert-block deduplication behaving differently on Replicated tables.