Discussion: Prometheus Rule Configurations
Today all rules of Prometheus are inside of our runbooks repo. The prometheus-operator for Kubernetes uses an object type of PrometheusRule to configure rules. Since we are going to be running a mixed infrastructure for quite a while we need to way to take our existing rules and convert them into Kubernetes objects and keep these in sync with how our existing Prometheus objects work today.
It's also been mentioned that we are potentially taxing some of our Prometheus boxes unnecessarily due to the amount of rules for which may be lacking data due to how we've segregated our targets for specific Prometheus servers. Due to this we need to consider dividing up what rules are placed on what Prometheus servers. This will become much more important as we build infrastructure in GKE as we don't want Prometheus evaluating rules for which we won't have any data for.
This may be epic worthy, so consider the impact that this work entails.
Reference doc: https://docs.google.com/document/d/1oJcWY4U2lmPwBvY8qQPlP7msPc7ioVUlo3ZKkCp-i1g/edit