Make labels order independent
This makes labels order independent in a way similar to what the official Prometheus::Client now does: https://github.com/prometheus/client_ruby/blob/7d9d45f547ce56ccf06344ab089e67228ddb7431/lib/prometheus/client/data_stores/direct_file_store.rb#L169
It shouldn't matter what order the labels were in for every change to the metric's value or how the metrics were initialized. But since the values were written out in 2 arrays before it did.
This works around that by sorting the label names and values alphabetically by label name before writing out the key of the metric into the file.
I discovered this while working on gitlab-org/gitlab!69154 (merged) for gitlab-com/gl-infra/scalability#1099
Questions remaining:
-
Would this have been a problem for existing metrics that were initialized differently form the way they are counted?
I don't think in our codebase it would make a big difference. We haven't done a lot of initialization in the past, other than the ones in the
RequestsRackMiddleware. We should be running into the differently ordered labels there. But the source metric for these is hard to load in prometheus. These might indicate that some of the series remain empty. I don't think they should for that label combination, but there might be another reason those instances aren't receiving traffic. I'll look into that further. -
Do we want to fix this here? Or should I work around it in gitlab-org/gitlab!69154 (merged)? In my local environment, I was missing metrics, which is why I'd rather not just ignore the problem.