Skip to content

chore(metrics): Add improved rollups for sums/gauges

Ankit Bhatnagar requested to merge abhatnagar/improved-counter-rollups into main

When beginning to work on #2561 with !2510 (merged), it was noticed that our rollups do not currently support computing aggregates such as avg, max, min or count - which would be necessary to add support for grouping searched metrics. This MR adds in a new version of rollups for sum & gauge metrics which include the necessary support.

Note, these have been added in addition to the old ones to maintain backward compatibility while we rollout the grouping feature. Once settled, the old set can be deleted gracefully. More related work to follow with !2526 (merged) & !2510 (merged). The current set of tests around data-rollup have however been updated to use the new MVs.

Tested on a dev-cluster with:

➜  abhtngr-dev git:(main) ✗ kubectl get cluster abhtngr-dev -o json | jq .status.conditions
[
  {
    "lastTransitionTime": "2024-04-30T14:40:57Z",
    "message": "All components are in ready state",
    "observedGeneration": 1,
    "reason": "Success",
    "status": "True",
    "type": "Ready"
  }
]
➜  abhtngr-dev git:(main) ✗ kubectl get cluster abhtngr-dev -o json | jq .status.lastMigrationApplied
"0.3.0-9429c7e9"
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) show tables like '%v2%'

SHOW TABLES LIKE '%v2%'

Query id: 89acf51b-05ee-462e-82c7-42d3422a7228

┌─name────────────────────────┐
│ goose_db_version_v2         │
│ metrics_main_gauge_1d_v2    │
│ metrics_main_gauge_1h_v2    │
│ metrics_main_gauge_1m_v2    │
│ metrics_main_gauge_mv_1d_v2 │
│ metrics_main_gauge_mv_1h_v2 │
│ metrics_main_gauge_mv_1m_v2 │
│ metrics_main_sum_1d_v2      │
│ metrics_main_sum_1h_v2      │
│ metrics_main_sum_1m_v2      │
│ metrics_main_sum_mv_1d_v2   │
│ metrics_main_sum_mv_1h_v2   │
│ metrics_main_sum_mv_1m_v2   │
└─────────────────────────────┘

13 rows in set. Elapsed: 0.233 sec.
cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(IngestionTimestamp) from metrics_main_sum;

SELECT max(IngestionTimestamp)
FROM metrics_main_sum

Query id: 8d95ce9b-3f98-4e51-acc2-a95084f723f3

┌───────max(IngestionTimestamp)─┐
│ 2024-04-30 20:23:37.689635529 │
└───────────────────────────────┘

1 row in set. Elapsed: 0.285 sec. Processed 2.94 million rows, 23.55 MB (10.32 million rows/s., 82.57 MB/s.)

cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_sum_1m_v2;

SELECT max(AggTimeUnix)
FROM metrics_main_sum_1m_v2

Query id: a1139f69-22c5-4460-8420-9db519b48e64

┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 20:23:00.000000000 │
└───────────────────────────────┘

1 row in set. Elapsed: 0.234 sec. Processed 10.62 thousand rows, 84.92 KB (45.40 thousand rows/s., 363.18 KB/s.)

cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_sum_1h_v2;

SELECT max(AggTimeUnix)
FROM metrics_main_sum_1h_v2

Query id: 42b23791-cfdf-41ab-93a9-9d90e90a4563

┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 20:00:00.000000000 │
└───────────────────────────────┘

1 row in set. Elapsed: 0.252 sec.

cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_sum_1d_v2;

SELECT max(AggTimeUnix)
FROM metrics_main_sum_1d_v2

Query id: 14b81282-730d-441b-b7b7-60dd51e5ed7b

┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 00:00:00.000000000 │
└───────────────────────────────┘

1 row in set. Elapsed: 0.232 sec.

cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(IngestionTimestamp) from metrics_main_gauge;

SELECT max(IngestionTimestamp)
FROM metrics_main_gauge

Query id: 57b8a43e-3247-4c4e-9873-8c6a3cdabc3d

┌───────max(IngestionTimestamp)─┐
│ 2024-04-30 20:23:44.874821366 │
└───────────────────────────────┘

1 row in set. Elapsed: 0.258 sec. Processed 403.36 thousand rows, 3.23 MB (1.56 million rows/s., 12.49 MB/s.)

cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_gauge_1m_v2;

SELECT max(AggTimeUnix)
FROM metrics_main_gauge_1m_v2

Query id: ecd46b0c-12ed-4af1-a411-55e6ee8ed9e6

┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 20:23:00.000000000 │
└───────────────────────────────┘

1 row in set. Elapsed: 0.233 sec. Processed 2.20 thousand rows, 17.58 KB (9.44 thousand rows/s., 75.51 KB/s.)

cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_gauge_1h_v2;

SELECT max(AggTimeUnix)
FROM metrics_main_gauge_1h_v2

Query id: a55acbe7-1482-4e55-b7d8-7cbb9ce6d20a

┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 20:00:00.000000000 │
└───────────────────────────────┘

1 row in set. Elapsed: 0.236 sec.

cluster-0-1-0.cluster-0-1.default.svc.cluster.local :) select max(AggTimeUnix) from metrics_main_gauge_1d_v2;

SELECT max(AggTimeUnix)
FROM metrics_main_gauge_1d_v2

Query id: 960bbdb1-8647-4c7c-a9a5-8d96948b232b

┌──────────────max(AggTimeUnix)─┐
│ 2024-04-30 00:00:00.000000000 │
└───────────────────────────────┘

1 row in set. Elapsed: 0.238 sec.

Related #2561.

Edited by Ankit Bhatnagar

Merge request reports