Audit metrics sending in old timestamps

We recently enabled an early compaction feature in Mimir that helps compact the TSDB Head to reduce the overall memory consumption of Ingesters during windows of high deployment/rollout activity.

However with this enabled we ran into an issue where our regional gprd cluster gprd-gitlab-gke started to get batches rejected while an early compaction was taking place.

Resulting in the error:

TSDB Head early compaction in progress and the write request contains samples overlapping with it

Any metric found that conflicts here cause the whole batch to be rejected, and because Mimir responds with a 503, Prometheus attempts to resend these.

The main driver to this is that we have a out_of_order_time_window set to 2hours on our main production tenant. This was set as we had quite a few metric sources sending through metrics with fairly old timesteamps.

These old timestamps are coming from exporters that expose their own timestamps, rather than having Prometheus set them at scrape time.

We need to audit what targets/exporters are exposing their own timestamps, and isolate which ones are presenting timestamps that are old.

Finally we need to decide on what an acceptable timestamp age is for our needs.