Skip to content

Support custom mappings for Prometheus integrations

Sarah Yasonik requested to merge sy-add-custom-mappings-for-prometheus into master

What does this MR do and why?

⚠ This MR is dependent on Add support for Prometheus as HTTP alert integr... (!120046 - merged). Add support for Prometheus as HTTP alert integr... (!120046 - merged) should merge first. This MR is also not user-facing yet.

Related issue: Allow Prometheus' metrics dashboard and incomin... (#338838)

Summary

  • This MR is preparation for migrating Integrations::Prometheus records to the table for AlertManagement::HttpIntegration.
  • There are no user-facing changes yet, but this does add support for custom mappings on Prometheus integrations. Once these are available to be saved via API, they will be broadly available.
  • This change is to enable us to separate alerting from metrics, so we can remove the Metrics Dashboard feature in %16.0. Custom mappings aren't strictly required for that removal, but it seems easier to just support them.

Changes in this MR

  • Refactor custom mappings logic into module
    • Also removed the coupling from the generic HTTP alerts path definitions
  • Switch to listing the integration name as the source of alerts in system notes
    • Because HTTP-backed integrations can be renamed, there's more value in preferring the integration name to the static default

Why no feature flag?

I initially started development on this issue with a feature flag, but the diff was very convoluted. So for this change, I think it's safer to have a clean diff, clear expectations of behavior, and keep backwards compatibility without a flag.

What about all the other stuff this needs?

Screenshots or screen recordings

Mapped payload in alert Updated system note source
Screenshot_2023-05-09_at_7.06.09_PM Screenshot_2023-05-09_at_7.06.14_PM

How to set up and validate locally

Pre-req: check out the branch & run migrations (added in !120046 (merged))

  1. Create an alert management HTTP integration with custom mapping
    • Via the UI (project > Settings > Monitor > expand Alerts > Add new integration)
    • Under Customize alert payload mapping, enter a sample of in individual alert from a proemtheus payload (under the alerts key) 
      {
  "startsAt":"2021-03-17T11:22:40Z",
  "endsAt":"2025-03-17T11:22:40Z",
  "generatorURL":"http://localhost:9090/graph?g0.expr=vector%281%29",
  "annotations":{
    "title":"default title",
    "description":"default description",
    "mapped_title":"mapped title",
    "mapped_description":"mapped description",
    "mapped_hosts":["mapped-host"],
    "mapped_fingerprint":"mapped fingerprint",
    "mapped_service":"mapped service",
    "mapped_monitoring_tool":"mapped monitoring tool",
    "mapped_severity":"high",
    "mapped_environment":"staging",
    "mapped_start":"2021-12-20T11:22:40Z",
    "mapped_end":"2024-12-20T11:22:40Z"
  },
  "labels":{
    "severity":"low",
    "gitlab_environment_name":"production"
  }
}
  • Select whichever fields you want to map to whichever available values 🤷
  1. Change the type on the integration to prometheus 
    • $ rails c

> AlertManagement::HttpIntegration.last.update!(type_identifier: :prometheus)
  2. Send a test alert via the UI
    • Sample payload: 
      {
  "alerts": [{
    "startsAt":"2021-03-17T11:22:40Z",
    "endsAt":"2025-03-17T11:22:40Z",
    "generatorURL":"http://localhost:9090/graph?g0.expr=vector%281%29",
    "annotations":{
      "title":"default title",
      "description":"default description",
      "mapped_title":"mapped title",
      "mapped_description":"mapped description",
      "mapped_hosts":["mapped-host"],
      "mapped_fingerprint":"mapped fingerprint",
      "mapped_service":"mapped service",
      "mapped_monitoring_tool":"mapped monitoring tool",
      "mapped_severity":"high",
      "mapped_environment":"staging",
      "mapped_start":"2021-12-20T11:22:40Z",
      "mapped_end":"2024-12-20T11:22:40Z"
    },
    "labels":{
      "severity":"low",
      "gitlab_environment_name":"production"
    }
  }],
  "status": "firing",
  "version": "4",
  "groupKey": null,
  "receiver": "",
  "externalURL": "",
  "groupLabels": {},
  "commonLabels": {},
  "commonAnnotations": {}
}
  3. See the new alert under project > Monitor > Alerts & check that the attributes correspond to the mapping

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sarah Yasonik

Merge request reports