Spike: Investigate Having GitLab Monitor Alerts on External Prometheus
Problem to solve
In #9409 we identified the problem where we do not support the ability to set alerts within GitLab, when a manually deployed Prometheus server is used.
This issue is a spike to investigate one of the two proposed solutions identified in that issue.
Further details
Specifically, we want to understand what the impact is if we use GitLab to monitor the alert, triggering a Prometheus query every X seconds (this is what Grafana does). As noted in that issue, there are concerns around the performance impact of doing this, especially if it adds increased load on the Prometheus server.
Questions that were raised:
- What is the increased load on Prometheus & Gitlab - could we potentially break the systems?
- If yes, can we scope down the types of available alerts to configure on Prometheus (maybe focus on several metrics)?
Proposal / Spike Details
Test what would happen if we were to use GitLab to monitor the alerts. Measure the impact on Prometheus with having GitLab query Prometheus every X seconds for X number of alerts.
Report on the impact in this issue so we can make an informed decision on #9409
Links / references
Original issue that this will inform: #9409