Alert on overloaded managed Prometheus server
As part of https://gitlab.com/gitlab-org/gitlab-ce/issues/28916, we will support deploying a managed Prometheus instance to a cluster.
As of right now, we do not set limits (or requests) on CPU and Memory, however there still may be natural scaling challenges for very large applications / clusters.
We should consider baking in alerts to the Prometheus server, utilizing its own self-instrumentation, to determine if it is being overloaded. This way GitLab can either present that data to the end-user so they can take action.
This mean launching a project specific, or environment specific Prometheus server, or perhaps reduce the cardinality of some of the metrics in the event a programming error led to a significant increase.
In the future, we could even consider a method to attempt to automatically rebalance, but providing the alert seems like the MVC here.