Default Prometheus Alerts are not applied to prometheus-server in gitlab-managed-app
Problem
GitLab creates default prometheus alerts via Prometheus::CreateDefaultAlertsService
with these parameters.
DEFAULT_ALERTS = [
{
identifier: 'response_metrics_nginx_ingress_16_http_error_rate',
operator: 'gt',
threshold: 0.1
},
{
identifier: 'response_metrics_nginx_ingress_http_error_rate',
operator: 'gt',
threshold: 0.1
},
{
identifier: 'response_metrics_nginx_http_error_percentage',
operator: 'gt',
threshold: 0.1
}
].freeze
It actually persists the objects PrometheusAlert
into database, however, it doesn't apply the alerting rules to the actual prometheus-server
in gitlab-managed-app
namespace on cluster. This is confusing as users are seeing alerts on UI, but it'll never be triggered.
Workaround
If user creates/updates/deletes at least one alert, the default values are also applied.
Edited by Shinya Maeda