Generic Alert Endpoint MVC
Problem to solve
Customers have many different tools they use to monitor their stack beyond Prometheus. We need to consume alerts from other sources. Since there are many monitoring tools that we could choose to integrate with, it would make more sense to create an endpoint/receiver/portal that customers can point their tools at a general portal that accepts alert payload.
Intended users
-
Delaney, Development Team Lead, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead
-
Sasha, Software Developer, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer
-
Devon, DevOps Engineer, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer
-
Sidney, Systems Administrator, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator
-
Sam, Security Analyst, https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sam-security-analyst
Further Details
This supports the Incident Management vision.
Original Proposal
Create a REST endpoint that accepts payloads and creates an issue with the payload in the body of the issue. Designate fields that users can send in with the payload that map to the title of the issue and the summary. The summary is the section at the top of all issues that are created automatically from alerts. The summary includes:
-
title
- This will be the title of the issue -
starts_at
- time the incident was triggered -
description
- high-level summary of the problem -
service
- affected service -
runbook
- associated run book
All of these fields will be optional from the user's perspective. If the user does not include the title
field in the payload of the incoming alert, we need to figure out what the default title will be.
If the user include any of the other fields (starts_at
, description
, service
, runbook
) in the payload of the incoming alert, we will exclude the field from the summary.
For the MVC, we will hard-code these fields and expect customers to configure alerts with values for these. Future iterations may include the ability for customers to choose what these fields are.
Questions:
- What format do the payloads need to be?
Working MVC proposal
- Receive alerts in the form of .json payloads to an alert endpoint (
<project-path>/alerts/notify.json
) asPOST
http method - GitLab will look for a field called
title
in the incoming .json. If it is not present, we will set the title of the issue to beNew: Incident
before creation and immediately following creation we will change it tooNew: Incident 123
- this will results in a system message posting to the incident timeline. - GitLab will also look for the following fields in the payload and paste them in the Summary section in the description of the issue in the order they are found in the payload. In any of the fields are not present, they will be excluded:
- start_time
- description
- monitoring_tool
- service
- hosts
- The entire payload will be posted in the issue discussion as a comment authored by the GitLab Alert Bot
- Configuration for this endpoint will live under Settings > Integrations > Alert Endpoint (this will be a new option in the integrations list).
Designs
Designs showing how the payload would look if sent to the issue comments can be seen in Sketch Cloud.
Authentication will take place in Settings > Integrations, where it will live within the project services list. The copy for the 'Alert Endpoint' list item can be see here. Clicking on that list item, will take the user to an alert endpoint configuration page.
Permissions and Security
Documentation
Documentation required.
Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Links / references
Current MRs
- Create a blank endpoint for generic alerts !15802 (merged)
- Add Settings page frontend !15733 (merged)
- Add generic alerts service & update project model - WIP !16117 (merged)
- Create issue from alert - !16021 (merged)
- Implement token-based authorization - #13203 (closed)
- Docs & Link to them from settings page - from !15733 (comment 213289744)
- Inline extra payload params !16441 (merged)
- Call
NotifyService
fromNotificationsController
!16508 (merged) - Split alert parameters for the summary and details boxes #31883 (closed) !18261 (merged)
- Check Generic Alert payload size !16911 (closed)
- Add content-type to docs !17606 (merged)
- Fetch payload body from params[:notification] !17673 (merged)
- Skip CSRF protection in Generic Alerts Endpoint !17692 (merged)
- Add proper token check for Notify Service !17759 (merged)
- Remove feature flag !18570 (merged)
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.