Generic Alert Endpoint MVC
Problem to solve
Customers have many different tools they use to monitor their stack beyond Prometheus. We need to consume alerts from other sources. Since there are many monitoring tools that we could choose to integrate with, it would make more sense to create an endpoint/receiver/portal that customers can point their tools at a general portal that accepts alert payload.
Delaney, Development Team Lead, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead
Sasha, Software Developer, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer
Sidney, Systems Administrator, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator
This supports the Incident Management vision.
Create a REST endpoint that accepts payloads and creates an issue with the payload in the body of the issue. Designate fields that users can send in with the payload that map to the title of the issue and the summary. The summary is the section at the top of all issues that are created automatically from alerts. The summary includes:
title- This will be the title of the issue
starts_at- time the incident was triggered
description- high-level summary of the problem
service- affected service
runbook- associated run book
All of these fields will be optional from the user's perspective. If the user does not include the
title field in the payload of the incoming alert, we need to figure out what the default title will be.
If the user include any of the other fields (
runbook) in the payload of the incoming alert, we will exclude the field from the summary.
For the MVC, we will hard-code these fields and expect customers to configure alerts with values for these. Future iterations may include the ability for customers to choose what these fields are.
- What format do the payloads need to be?
Working MVC proposal
- Receive alerts in the form of .json payloads to an alert endpoint (
- GitLab will look for a field called
titlein the incoming .json. If it is not present, we will set the title of the issue to be
New: Incidentbefore creation and immediately following creation we will change it too
New: Incident 123- this will results in a system message posting to the incident timeline.
- GitLab will also look for the following fields in the payload and paste them in the Summary section in the description of the issue in the order they are found in the payload. In any of the fields are not present, they will be excluded:
- The entire payload will be posted in the issue discussion as a comment authored by the GitLab Alert Bot
- Configuration for this endpoint will live under Settings > Integrations > Alert Endpoint (this will be a new option in the integrations list).
Designs showing how the payload would look if sent to the issue comments can be seen in Sketch Cloud.
Authentication will take place in Settings > Integrations, where it will live within the project services list. The copy for the 'Alert Endpoint' list item can be see here. Clicking on that list item, will take the user to an alert endpoint configuration page.
Permissions and Security
What does success look like, and how can we measure that?
What is the type of buyer?
Links / references
- Create a blank endpoint for generic alerts !15802 (merged)
- Add Settings page frontend !15733 (merged)
- Add generic alerts service & update project model - WIP !16117 (merged)
- Create issue from alert - !16021 (merged)
- Implement token-based authorization - #13203
- Docs & Link to them from settings page - from !15733 (comment 213289744)
- Inline extra payload params !16441 (merged)
- Split alert parameters for the summary and details boxes #31883 (closed) !18261 (merged)
- Check Generic Alert payload size !16911 (closed)
- Add content-type to docs !17606 (merged)
- Fetch payload body from params[:notification] !17673 (merged)
- Skip CSRF protection in Generic Alerts Endpoint !17692 (merged)
- Add proper token check for Notify Service !17759 (merged)
- Remove feature flag !18570 (merged)/