Automatically add Gitlab webhook to internal Alertmanager config

Problem to solve

We need to set a webhook in alertmanager config pointing to an endpoint of the self-monitoring project in the Rails app. For the webhook, we need two pieces of data; a token and the API URL.

1. API URL
   The API URL is of the form http(s)://:host/:namespace/:project/prometheus/alerts/notify.json. We will have to hardcode the project and namespace in the URL since the project will not have been created as yet.

2. Token
   The token is used to match alert notifications. So, if an alert is received by GitLab with a token that is not present in DB, the alert is ignored. We can generate a token in Omnibus and place it in the alertmanager config and in the gitlab.yml file. Then the SelfMonitoring::Project::CreateService will pick up the token from the gitlab.yml file and save it into the DB (project.alerting_setting.token) so that incoming alerts are accepted.

This will only work for clients using the internal Prometheus server.

Intended users

Further details

Proposal

Permissions and Security

Documentation

Testing

What does success look like, and how can we measure that?

Links / references