Automatically add Gitlab webhook to internal Alertmanager config
Problem to solve
We need to set a webhook in alertmanager config pointing to an endpoint of the self-monitoring project in the Rails app. For the webhook, we need two pieces of data; a token and the API URL.
-
API URL
The API URL is of the formhttp(s)://:host/:namespace/:project/prometheus/alerts/notify.json
. We will have to hardcode the project and namespace in the URL since the project will not have been created as yet. -
Token
The token is used to match alert notifications. So, if an alert is received by GitLab with a token that is not present in DB, the alert is ignored. We can generate a token in Omnibus and place it in the alertmanager config and in the gitlab.yml file. Then the SelfMonitoring::Project::CreateService will pick up the token from the gitlab.yml file and save it into the DB (project.alerting_setting.token
) so that incoming alerts are accepted.
This will only work for clients using the internal Prometheus server.