refactor(server): replace QueryProfiler with log_comment-based system table queries

Michael Usachenkorequested to merge
graphsec/profiler-simplification into main

Problem

The QueryProfiler in clickhouse-client maintains a parallel reqwest-based HTTP client (~486 lines) that reimplements URL param building, auth headers, and Arrow-IPC parsing. All of this exists because clickhouse-rs discards X-ClickHouse-Summary response headers, so the profiler needed wait_end_of_query=1 to capture read_rows, read_bytes, memory_usage synchronously.

!768 (merged) added log_comment to every ClickHouse query. That means every query is now identifiable in system.query_log. The profiler's custom HTTP client is no longer necessary -- we can run queries through the normal client and look up the stats afterward.

The two separate execution paths (execute_standard vs execute_profiled) in the server pipeline also made the code harder to follow. Every change to query execution had to be made in two places.

Solution

Delete profiler.rs entirely. Replace it with profiling.rs -- methods on ArrowClickHouseClient that query system tables using the standard clickhouse-rs client.

Single execution path. All queries go through the normal clickhouse-rs client. When profiling is enabled, a UUID profiling_id is appended to log_comment. After execution, SYSTEM FLUSH LOGS + a system.query_log lookup by that profiling_id retrieves the same stats the old X-ClickHouse-Summary headers provided. EXPLAIN queries and system.processors_profile_log lookups also go through the normal client.

Dependency removal. reqwest and once_cell are dropped from clickhouse-client. ClickHouseError::Http variant is removed.

References

Edited by Michael Usachenko

Merge request reports