Loading
Add query_log_usage table and materialized view
What does this MR do and why?
Add query_log_usage table and materialized view where fully instrumented query metrics will be collected.
Related to: https://gitlab.com/groups/gitlab-com/gl-infra/finops/-/work_items/129+
| Column | Description |
|---|---|
event_time |
Query finish time, UTC. Partition key and leading sort key. |
query_id |
ClickHouse query UUID. Second sort key. |
root_namespace_id |
Root namespace, from log_comment. Rows without one are dropped. |
organization_id |
Organization, from log_comment. 0 when absent. |
user_id |
User who triggered the query. 0 when absent. |
correlation_id |
Request correlation id. Joins back to the Rails logs. |
application |
web, sidekiq, console or test. |
feature_category |
GitLab feature category. '' when absent. |
query_kind |
Select, Insert, Create, Drop, etc. |
databases |
Databases the query touched. |
tables |
Tables touched. |
query_duration_ms |
Wall-clock duration. |
read_rows |
Rows scanned. |
read_bytes |
Bytes scanned. |
memory_usage_bytes |
Peak memory. |
os_cpu_virtual_time_us |
CPU time, from ProfileEvents['OSCPUVirtualTimeMicroseconds']. |
exception_code |
0 on success, else the ClickHouse error code. |
log_comment |
The raw comment, kept for keys we don't extract yet. |
vcpu_seconds |
ALIAS. os_cpu_virtual_time_us / 1e6. |
memory_gb_seconds |
ALIAS. memory_usage_bytes * query_duration_ms / 1e12. Upper bound. |
References
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Adam Hegyi