chore(metrics): Add improved rollups for sums/gauges
When beginning to work on #2561 with !2510 (merged), it was noticed that our rollups do not currently support computing aggregates such as avg
, max
, min
or count
- which would be necessary to add support for grouping searched metrics. This MR adds in a new version of rollups for sum
& gauge
metrics which include the necessary support.
Note, these have been added in addition to the old ones to maintain backward compatibility while we rollout the grouping feature. Once settled, the old set can be deleted gracefully. More related work to follow with !2526 (merged) & !2510 (merged). The current set of tests around data-rollup have however been updated to use the new MVs.
Tested on a dev-cluster with:
➜ abhtngr-dev git:(main) ✗ kubectl get cluster abhtngr-dev -o json | jq .status.conditions
[
{
"lastTransitionTime": "2024-04-30T14:40:57Z",
"message": "All components are in ready state",
"observedGeneration": 1,
"reason": "Success",
"status": "True",
"type": "Ready"
}
]
➜ abhtngr-dev git:(main) ✗ kubectl get cluster abhtngr-dev -o json | jq .status.lastMigrationApplied
"0.3.0-9429c7e9"
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) show tables like '%v2%'
SHOW TABLES LIKE '%v2%'
Query id: 89acf51b-05ee-462e-82c7-42d3422a7228
┌─name────────────────────────┐
│ goose_db_version_v2 │
│ metrics_main_gauge_1d_v2 │
│ metrics_main_gauge_1h_v2 │
│ metrics_main_gauge_1m_v2 │
│ metrics_main_gauge_mv_1d_v2 │
│ metrics_main_gauge_mv_1h_v2 │
│ metrics_main_gauge_mv_1m_v2 │
│ metrics_main_sum_1d_v2 │
│ metrics_main_sum_1h_v2 │
│ metrics_main_sum_1m_v2 │
│ metrics_main_sum_mv_1d_v2 │
│ metrics_main_sum_mv_1h_v2 │
│ metrics_main_sum_mv_1m_v2 │
└─────────────────────────────┘
13 rows in set. Elapsed: 0.233 sec.
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(IngestionTimestamp) from metrics_main_sum;
SELECT max(IngestionTimestamp)
FROM metrics_main_sum
Query id: 8d95ce9b-3f98-4e51-acc2-a95084f723f3
┌───────max(IngestionTimestamp)─┐
│ 2024-04-30 20:23:37.689635529 │
└───────────────────────────────┘
1 row in set. Elapsed: 0.285 sec. Processed 2.94 million rows, 23.55 MB (10.32 million rows/s., 82.57 MB/s.)
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_sum_1m_v2;
SELECT max(AggTimeUnix)
FROM metrics_main_sum_1m_v2
Query id: a1139f69-22c5-4460-8420-9db519b48e64
┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 20:23:00.000000000 │
└───────────────────────────────┘
1 row in set. Elapsed: 0.234 sec. Processed 10.62 thousand rows, 84.92 KB (45.40 thousand rows/s., 363.18 KB/s.)
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_sum_1h_v2;
SELECT max(AggTimeUnix)
FROM metrics_main_sum_1h_v2
Query id: 42b23791-cfdf-41ab-93a9-9d90e90a4563
┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 20:00:00.000000000 │
└───────────────────────────────┘
1 row in set. Elapsed: 0.252 sec.
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_sum_1d_v2;
SELECT max(AggTimeUnix)
FROM metrics_main_sum_1d_v2
Query id: 14b81282-730d-441b-b7b7-60dd51e5ed7b
┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 00:00:00.000000000 │
└───────────────────────────────┘
1 row in set. Elapsed: 0.232 sec.
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(IngestionTimestamp) from metrics_main_gauge;
SELECT max(IngestionTimestamp)
FROM metrics_main_gauge
Query id: 57b8a43e-3247-4c4e-9873-8c6a3cdabc3d
┌───────max(IngestionTimestamp)─┐
│ 2024-04-30 20:23:44.874821366 │
└───────────────────────────────┘
1 row in set. Elapsed: 0.258 sec. Processed 403.36 thousand rows, 3.23 MB (1.56 million rows/s., 12.49 MB/s.)
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_gauge_1m_v2;
SELECT max(AggTimeUnix)
FROM metrics_main_gauge_1m_v2
Query id: ecd46b0c-12ed-4af1-a411-55e6ee8ed9e6
┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 20:23:00.000000000 │
└───────────────────────────────┘
1 row in set. Elapsed: 0.233 sec. Processed 2.20 thousand rows, 17.58 KB (9.44 thousand rows/s., 75.51 KB/s.)
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_gauge_1h_v2;
SELECT max(AggTimeUnix)
FROM metrics_main_gauge_1h_v2
Query id: a55acbe7-1482-4e55-b7d8-7cbb9ce6d20a
┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 20:00:00.000000000 │
└───────────────────────────────┘
1 row in set. Elapsed: 0.236 sec.
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_gauge_1d_v2;
SELECT max(AggTimeUnix)
FROM metrics_main_gauge_1d_v2
Query id: 960bbdb1-8647-4c7c-a9a5-8d96948b232b
┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 00:00:00.000000000 │
└───────────────────────────────┘
1 row in set. Elapsed: 0.238 sec.
Related #2561.
Edited by Ankit Bhatnagar