Migration sometimes fails with "error migrating time step: v2Ref non-empty; unable to add to v2 storage: unknown series: not found"
Thanks for this tool!
We have successfully migrated Prometheus v1 storage using this tool for up to 4000h retention, however when running it on a 10000h retention storage (default step, varying parallelism doesn't seem to make a difference) we're receiving this error:
level=error err="error migrating storage: error migrating time step: v2Ref non-empty; unable to add to v2 storage: unknown series: not found"
We've retried the same migration (after deleting data-v2
) multiple times and the error seems to appear at random progress during the process, however reducing lookback to 8500h we were able to successfully complete a migration. Moreover the v1 storage contains some metrics in orphaned
directory, in case that's relevant.
I'm happy to help debug this further, reporting it here in case other people run into the same.