Allow generic endpoint to receive alerts from external Prometheus

Problem to solve

In order to setup Alert Management for a external Prometheus instance users have to complete the following steps:

1. Enable Alert Management via Settings > Integration > Alerts endpoint
2. Enable and configure manual Prometheus via Settings > Integrations > Prometheus
3. Use the endpoint URL from Prometheus integration page and not from Alerts endpoint page to configure their Prometheus' Alertmanager instance

It's very easy to miss step 3 and use the "wrong" token.

If Prometheus payload is sent to the Generic JSON endpoint GitLab creates an Alert with less useful Alert details. In contrast, Prometheus payload sent to the Prometheus JSON endpoint is parsed and shown properly.

Generic JSON endpoint	Prometheus endpoint

Ideally, we'd like to improve user experience and make step 2 and 3 unnecessary.

Intended users

• Devon (DevOps Engineer)
• Sidney (Systems Administrator)
• Allison (Application Ops)

User experience goal

Don't require users to remember step 2 and 3.

Proposal

Allow generic endpoint to receive alerts from external Prometheus:

• dispatch to Prometheus specific notification service in the generic endpoint controller if the payload is Prometheus-specific
• let Prometheus specific notification service validate tokens configured via Settings > Operations > Alerts endpoint
• document this behavior in Alert Management and Prometheus integration docs

Further details

Permissions and Security

Handling Prometheus payload is guarded by the generated alert manager tokens which is not associated with a GitLab user.

Documentation

Availability & Testing

We can extend existing unit and integration specs to test the new feature.

What is the type of buyer?

GitLab Core since both Alert Management and Prometheus integration is GitLab Core.

Links / references

See for the discussion: &3360 (comment 345755346)