fix(query): remove server-side system.query_log enrichment

Problem

The query pipeline had an enrich_execution path that, when ProfilingConfig was enabled, attempted to call SYSTEM FLUSH LOGS and then read from system.query_log to backfill detailed execution stats (read_rows, read_bytes, memory, ProfileEvents, EXPLAIN plans, processor profiles) into the query response. The hydration stage had an equivalent block.

In production, the webserver runs as gkg_reader -- a role with SELECT ON gkg.* only. It cannot execute SYSTEM FLUSH LOGS (requires the SYSTEM privilege) and cannot read system.query_log (outside the gkg database grant). Every enrichment attempt fails at the first statement.

The failure was invisible. Both call sites used if let Ok(Some(entry)) = ..., silently discarding the permission error. EXPLAIN calls used .ok(). No log line, no metric, no error in the response. If someone enabled profiling on a deployed server, they would see empty profiling fields and no indication of why.

Solution

The enrich_execution codepath has been removed. The codepaths involving the X-ClickHouse-Summary header path are untouched -- it requires no special privileges and continues to provide baseline stats (read_rows, read_bytes, memory_usage) on every query.

Why this is a separation of concerns issue

Profiling and serving are fundamentally different responsibilities with different trust and privilege requirements:

  • Serving needs the minimum privileges to execute compiled queries and return results. The gkg_reader role is correctly scoped for this.
  • Profiling needs elevated access to ClickHouse internals: system tables, SYSTEM commands, EXPLAIN plans, processor-level diagnostics.

Mixing both into the same code path means the server binary either needs privileges it shouldn't have, or the profiling code is dead. We had the latter, dressed up as a config toggle that could never actually work in the deployed environment.

The profiler CLI (crates/query-engine/profiler/) already exists as the dedicated tool for this. It runs with explicit credentials, fetches the same system.query_log data, supports EXPLAIN plans and processor profiles, and does comparative benchmarking. It is purpose-built for the elevated-privilege workflow that the server pipeline was silently failing at.

Edited by Michael Usachenko

Merge request reports

Loading