Set alerts for manually deployed Prometheus servers
Problem to solve
Currently we do not support the ability to set alerts within GitLab, when a manually deployed Prometheus server is used.
This leads to primarily two problems:
- Correlating an incoming alert to a specific or chart or metric is impossible without manual association in some way (e.g. setting a "chart identifier" in the alert)
- Managing alerts requires editing YML files on the Prometheus box, which adds significant overhead when using the GitLab dashboard
Target audience
DevOps, Ops
Further details
Proposal
We could solve this one of two ways:
- Utilize GitLab to monitor the alert, triggering a Prometheus query every X seconds (this is what Grafana does)
- A GitLab helper utility installed on the Prometheus server, which polls the GitLab server for alerts and manages an alert config file. This effectively replicates what we do with the managed servers.
GitLab monitors the alert
Pros:
- No additional configuration required on behalf of the user
- Utilizes the existing API query mechanism we've already built
Cons:
- Increased load on GitLab
- Increased load on Prometheus
GitLab Helper Utility to manage alert config
Pros:
- No increased load on GitLab
- Reduced load on Prometheus
Cons:
- Needs to be installed and configured
What does success look like, and how can we measure that?
Ability to set an alert for a manually deployed Prometheus server as easy as you can for an automatically managed GitLab Prometheus server.