Successful migration results in disjoint metrics with same labelset
Hi,
I am running into a problem with prometheus-storage-migrator where the migrated metrics have the same labels as new metrics, but are regarded by Prometheus as two different metrics. This didn't use to be a problem, however since Prometheus 2.5 it isn't possible to run operations like rate on such results (cfr https://github.com/prometheus/prometheus/pull/4589 and https://github.com/prometheus/prometheus/issues/4562), which ends up breaking dashboards and alerts that cover time spans with old and new metrics.
A query_range query for a single metric including pre and post migration looks like the output below (kubernetes_build_info{instance="10.192.0.93:6443"}).
Any ideas/pointers on where/how to debug this issue further? I'm happy to share privately the underlying storage too.
thanks!
$ curl -s 'http://localhost:9906/k8s/api/v1/query_range?query=kubernetes_build_info%7Binstance%3D%2210.192.0.93%3A6443%22%7D&start=1551605804.957&end=1552469804.957&step=3456&_=1552406950990' | jq
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"__name__": "kubernetes_build_info",
"buildDate": "2018-12-04T16:25:24Z",
"compiler": "gc",
"gitCommit": "637c7e288581ee40ab4ca210618a89a555b6e7e9",
"gitTreeState": "archive",
"gitVersion": "v1.10.11",
"goVersion": "go1.9.3",
"instance": "10.192.0.93:6443",
"job": "k8s-api",
"major": "1",
"minor": "10",
"platform": "linux/amd64"
},
"values": [
[
1552469804.957,
"1"
]
]
},
{
"metric": {
"__name__": "kubernetes_build_info",
"buildDate": "2018-12-04T16:25:24Z",
"compiler": "gc",
"gitCommit": "637c7e288581ee40ab4ca210618a89a555b6e7e9",
"gitTreeState": "archive",
"gitVersion": "v1.10.11",
"goVersion": "go1.9.3",
"instance": "10.192.0.93:6443",
"job": "k8s-api",
"major": "1",
"minor": "10",
"platform": "linux/amd64"
},
"values": [
[
1551605804.957,
"1"
],
[
1551609260.957,
"1"
],
[
1551612716.957,
"1"
],
[
1551616172.957,
"1"
],
[
1551619628.957,
"1"
],
[
1551623084.957,
"1"
],
[
1551626540.957,
"1"
],
[
1551629996.957,
"1"
],
[
1551633452.957,
"1"
],
[
1551636908.957,
"1"
],
[
1551640364.957,
"1"
],
[
1551643820.957,
"1"
],
[
1551647276.957,
"1"
],
[
1551650732.957,
"1"
],
[
1551654188.957,
"1"
],
[
1551657644.957,
"1"
],
[
1551661100.957,
"1"
],
[
1551664556.957,
"1"
],
[
1551668012.957,
"1"
],
[
1551671468.957,
"1"
],
[
1551674924.957,
"1"
],
[
1551678380.957,
"1"
],
[
1551681836.957,
"1"
],
[
1551685292.957,
"1"
]
]
}
]
}
}