Skip to content

Successful migration results in disjoint metrics with same labelset

Hi, I am running into a problem with prometheus-storage-migrator where the migrated metrics have the same labels as new metrics, but are regarded by Prometheus as two different metrics. This didn't use to be a problem, however since Prometheus 2.5 it isn't possible to run operations like rate on such results (cfr https://github.com/prometheus/prometheus/pull/4589 and https://github.com/prometheus/prometheus/issues/4562), which ends up breaking dashboards and alerts that cover time spans with old and new metrics.

A query_range query for a single metric including pre and post migration looks like the output below (kubernetes_build_info{instance="10.192.0.93:6443"}).

Any ideas/pointers on where/how to debug this issue further? I'm happy to share privately the underlying storage too.

thanks!

$ curl -s 'http://localhost:9906/k8s/api/v1/query_range?query=kubernetes_build_info%7Binstance%3D%2210.192.0.93%3A6443%22%7D&start=1551605804.957&end=1552469804.957&step=3456&_=1552406950990' | jq
{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "__name__": "kubernetes_build_info",
          "buildDate": "2018-12-04T16:25:24Z",
          "compiler": "gc",
          "gitCommit": "637c7e288581ee40ab4ca210618a89a555b6e7e9",
          "gitTreeState": "archive",
          "gitVersion": "v1.10.11",
          "goVersion": "go1.9.3",
          "instance": "10.192.0.93:6443",
          "job": "k8s-api",
          "major": "1",
          "minor": "10",
          "platform": "linux/amd64"
        },
        "values": [
          [
            1552469804.957,
            "1"
          ]
        ]
      },
      {
        "metric": {
          "__name__": "kubernetes_build_info",
          "buildDate": "2018-12-04T16:25:24Z",
          "compiler": "gc",
          "gitCommit": "637c7e288581ee40ab4ca210618a89a555b6e7e9",
          "gitTreeState": "archive",
          "gitVersion": "v1.10.11",
          "goVersion": "go1.9.3",
          "instance": "10.192.0.93:6443",
          "job": "k8s-api",
          "major": "1",
          "minor": "10",
          "platform": "linux/amd64"
        },
        "values": [
          [
            1551605804.957,
            "1"
          ],
          [
            1551609260.957,
            "1"
          ],
          [
            1551612716.957,
            "1"
          ],
          [
            1551616172.957,
            "1"
          ],
          [
            1551619628.957,
            "1"
          ],
          [
            1551623084.957,
            "1"
          ],
          [
            1551626540.957,
            "1"
          ],
          [
            1551629996.957,
            "1"
          ],
          [
            1551633452.957,
            "1"
          ],
          [
            1551636908.957,
            "1"
          ],
          [
            1551640364.957,
            "1"
          ],
          [
            1551643820.957,
            "1"
          ],
          [
            1551647276.957,
            "1"
          ],
          [
            1551650732.957,
            "1"
          ],
          [
            1551654188.957,
            "1"
          ],
          [
            1551657644.957,
            "1"
          ],
          [
            1551661100.957,
            "1"
          ],
          [
            1551664556.957,
            "1"
          ],
          [
            1551668012.957,
            "1"
          ],
          [
            1551671468.957,
            "1"
          ],
          [
            1551674924.957,
            "1"
          ],
          [
            1551678380.957,
            "1"
          ],
          [
            1551681836.957,
            "1"
          ],
          [
            1551685292.957,
            "1"
          ]
        ]
      }
    ]
  }
}