Work on Mimir cost optimization

With the move to remote-write we expect increased operating costs from Thanos as we are shipping data around and are running at a larger scale, so we can inject proper performance into our metric system.

However we are seeing much higher than expected inter-zone AZ traffic costs, see: https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/25040

Compression was enabled through gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!4267 (merged) which should help mitigate some of this cost, however we should drill down further and look into cost optimisation now that we have onboarded almost all of our metrics.

Things to look at:

• Should we consider running in a single AZ with failover options for a secondary AZ
• Spot instances could be a viable option for reduced resource costs
• Autoscaling of resources where possible (Issue open here)