Determine if adding offset can improve accuracy in present metrics.
Problem
We have created a new apdex methodthat requires three things in order to improve accuracy: recording from source metrics, a small offset, and an all or nothing transactional numerator and denominator.
The source metrics and the all or nothing query are not trivial to modify in our current recording rules, but the offset is. We have an ongoing issue with the accuracy of the error budget data, so this issue will explore the possibility of a short term fix for the problem while we continue with the longer term fix.
This experiment will check two things:
- Can adding an offset only to the thanos recording rules that calculate apdex improve accuracy?
- Can adding an offset for prometheus recording rules only improve accuracy?
- Can adding an offset at every level of the recording rules improve accuracy?
Solution
We have determined that adding a 30 second offset to the prometheus recording rules will improve our accuracy from mostly getting 3 digits of accuracy (three nines) with occasional drops to two (two nines) to mostly getting 4 digits of accuracy (four nines) with occasional drops to three (three nines).
We are implementing this change now in #2486 (closed) and #2482 (closed).