Add query_log_usage table and materialized view

What does this MR do and why?

Add query_log_usage table and materialized view where fully instrumented query metrics will be collected.

Related to: https://gitlab.com/groups/gitlab-com/gl-infra/finops/-/work_items/129+

Column Description
event_time Query finish time, UTC. Partition key and leading sort key.
query_id ClickHouse query UUID. Second sort key.
root_namespace_id Root namespace, from log_comment. Rows without one are dropped.
organization_id Organization, from log_comment. 0 when absent.
user_id User who triggered the query. 0 when absent.
correlation_id Request correlation id. Joins back to the Rails logs.
application web, sidekiq, console or test.
feature_category GitLab feature category. '' when absent.
query_kind Select, Insert, Create, Drop, etc.
databases Databases the query touched.
tables Tables touched.
query_duration_ms Wall-clock duration.
read_rows Rows scanned.
read_bytes Bytes scanned.
memory_usage_bytes Peak memory.
os_cpu_virtual_time_us CPU time, from ProfileEvents['OSCPUVirtualTimeMicroseconds'].
exception_code 0 on success, else the ClickHouse error code.
log_comment The raw comment, kept for keys we don't extract yet.
vcpu_seconds ALIAS. os_cpu_virtual_time_us / 1e6.
memory_gb_seconds ALIAS. memory_usage_bytes * query_duration_ms / 1e12. Upper bound.

References

Screenshots or screen recordings

Before After

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Adam Hegyi

Merge request reports

Loading
Loading