Skip to content
GitLab Next
Closed
Created by Sarah Waldner@sarahwaldner🍉Developer

Generic Alert Endpoint MVC

Problem to solve

Customers have many different tools they use to monitor their stack beyond Prometheus. We need to consume alerts from other sources. Since there are many monitoring tools that we could choose to integrate with, it would make more sense to create an endpoint/receiver/portal that customers can point their tools at a general portal that accepts alert payload.

Intended users

Further Details

This supports the Incident Management vision.

Original Proposal

Create a REST endpoint that accepts payloads and creates an issue with the payload in the body of the issue. Designate fields that users can send in with the payload that map to the title of the issue and the summary. The summary is the section at the top of all issues that are created automatically from alerts. The summary includes:

  • title - This will be the title of the issue
  • starts_at - time the incident was triggered
  • description - high-level summary of the problem
  • service - affected service
  • runbook - associated run book

All of these fields will be optional from the user's perspective. If the user does not include the title field in the payload of the incoming alert, we need to figure out what the default title will be. If the user include any of the other fields (starts_at, description, service, runbook) in the payload of the incoming alert, we will exclude the field from the summary.

For the MVC, we will hard-code these fields and expect customers to configure alerts with values for these. Future iterations may include the ability for customers to choose what these fields are.

Questions:

  • What format do the payloads need to be?

Working MVC proposal

  • Receive alerts in the form of .json payloads to an alert endpoint (<project-path>/alerts/notify.json) as POST http method
  • GitLab will look for a field called title in the incoming .json. If it is not present, we will set the title of the issue to be New: Incident before creation and immediately following creation we will change it too New: Incident 123 - this will results in a system message posting to the incident timeline.
  • GitLab will also look for the following fields in the payload and paste them in the Summary section in the description of the issue in the order they are found in the payload. In any of the fields are not present, they will be excluded:
    • start_time
    • description
    • monitoring_tool
    • service
    • hosts
  • The entire payload will be posted in the issue discussion as a comment authored by the GitLab Alert Bot
  • Configuration for this endpoint will live under Settings > Integrations > Alert Endpoint (this will be a new option in the integrations list).

Designs

Designs showing how the payload would look if sent to the issue comments can be seen in Sketch Cloud.

Authentication will take place in Settings > Integrations, where it will live within the project services list. The copy for the 'Alert Endpoint' list item can be see here. Clicking on that list item, will take the user to an alert endpoint configuration page.

Permissions and Security

Documentation

Documentation required.

Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Links / references

Current MRs

  1. Create a blank endpoint for generic alerts !15802 (merged)
  2. Add Settings page frontend !15733 (merged)
  3. Add generic alerts service & update project model - WIP !16117 (merged)
  4. Create issue from alert - !16021 (merged)
  5. Implement token-based authorization - #13203 (closed)
  6. Docs & Link to them from settings page - from !15733 (comment 213289744)
  7. Inline extra payload params !16441 (merged)
  8. Call NotifyService from NotificationsController !16508 (merged)
  9. Split alert parameters for the summary and details boxes #31883 (closed) !18261 (merged)
  10. Check Generic Alert payload size !16911 (closed)
  11. Add content-type to docs !17606 (merged)
  12. Fetch payload body from params[:notification] !17673 (merged)
  13. Skip CSRF protection in Generic Alerts Endpoint !17692 (merged)
  14. Add proper token check for Notify Service !17759 (merged)
  15. Remove feature flag !18570 (merged)
Edited by Vitali Tatarintev