Skip to content
GitLab Next
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • P prometheus-storage-migrator
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 2
    • Issues 2
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.org
  • prometheus-storage-migrator
  • Issues
  • #4
Closed
Open
Created Feb 26, 2019 by Filippo Giunchedi@filippogContributor

Migration sometimes fails with "error migrating time step: v2Ref non-empty; unable to add to v2 storage: unknown series: not found"

Thanks for this tool!

We have successfully migrated Prometheus v1 storage using this tool for up to 4000h retention, however when running it on a 10000h retention storage (default step, varying parallelism doesn't seem to make a difference) we're receiving this error:

level=error err="error migrating storage: error migrating time step: v2Ref non-empty; unable to add to v2 storage: unknown series: not found"

We've retried the same migration (after deleting data-v2) multiple times and the error seems to appear at random progress during the process, however reducing lookback to 8500h we were able to successfully complete a migration. Moreover the v1 storage contains some metrics in orphaned directory, in case that's relevant.

I'm happy to help debug this further, reporting it here in case other people run into the same.

Assignee
Assign to
Time tracking