At the time of writing up this issue, the stable/prometheus-operator helm chart does not support the same version of thanos that we are running throughout the rest of our infrastructure. The use of thanos is preferred to ensure the longevity of data beyond the time for which prometheus is able to store data in our GKE clusters. Utilize this issue to discuss/investigate/create actionable issues to ensure that we get thanos enabled with the desired configuration in GKE, and that prometheus is configured in a preferred manner.
- Check-up on which version the latest version of the stable/promethues-operator chart supports such that we can hopefully get the same version of thanos deployed across both our VM and Kubernetes infrastructure
Create a separate service account for this
- Currently the same service account used to mangle user data is storing metric data, this is not best security practice.
- Configure thanos to store metrics in the cloud
- Turn down the data retention of prometheus from 8 weeks to a single day