Validation of time series across Thanos and Mimir
This is following up with https://docs.google.com/document/d/13TW4x3ofw0RxifZvZ7eNvrPxFnnXmhzQ8fal3fhYgjg/edit#bookmark=id.vlmjz1lb9juy and a problem @hmerscher pointed out recently - how do we confidently compare data across Thanos and Mimir, for example to validate recording rules are set up correctly?
The problem is that time series from both systems won't match exactly, for various reasons.
I'm doing a timeboxed thing where I'd like to get a Jupyter notebook that allows us to try a few things:
- Load data from Thanos and Mimir API. This needs Expose Mimir endpoint internally and through IAP (#2875 - closed) for the Mimir part.
- Some juggling to make those comparable
- Derive a statistical measure to tell how similar they are.
One approach to (3) is to use Time Warping to see how close time series go together. I find this interesting because I understand most of the discrepancies (for otherwise identical time series) come from different sampling times - introducing a skew in the time axis.