Skip to content
Snippets Groups Projects

Stop returning alerts from prometheus alerts endpoint

All threads resolved!

What does this MR do and why?

Implements #372115 (closed)

Do not return any JSON response from the Prometheus alerts endpoint.

When Prometheus sends an alert it does not expect any returning values. We can safely drop the JSON response.

How to verify locally

Prometheus

  1. Setup [Prometheus Integration]https://docs.gitlab.com/ee/operations/incident_management/integrations.html#http-endpoints) via Settings > Monitor > Alerts
  2. Clone https://gitlab.com/splattael/gitlab-manual-prometheus
  3. Edit gitlab-manual-prometheus/alertmanager.yml:
global: {}
receivers:
- name: gitlab
  webhook_configs:
  - send_resolved: true
    # e.g. url: http://192.168.10.38:3001/manual-prometheus/autodevops-deploy/prometheus/alerts/notify.json
    url: http://localhost:3000/gitlab-org/gitlab-test/prometheus/alerts/notify.json
    http_config:
      # e.g. bearer_token: 2195dc9c136400428d0124805e78a7bb
      bearer_token: c34fc9a8840666ba815429a2af6dab94
route:
  group_interval: 5s
  group_wait: 5s
  receiver: gitlab
  repeat_interval: 5m
  1. gdk stop prometheus (if running) to allow our prometheus to be bind on port 9090
  2. docker-compose up and wait ~10 seconds
  3. See http://localhost:3000/gitlab-org/gitlab-test/-/alert_management/2/details
  4. Via curl :white_check_mark:
$ curl -v --request POST --data '{"version":"4","receiver":"gitlab","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"title","gitlab":"hook","gitlab_alert_id":""},"annotations":{},"startsAt":"2018-09-24T08:57:31.095725221Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://prometheus-prometheus-server-URL"}],"groupLabels":{"alertname":"title"},"commonLabels":{"alertname":"title","gitlab":"hook","gitlab_alert_id":null},"commonAnnotations":{},"externalURL":"","groupKey":"{}:{alertname='title'}"}
' --header "Authorization: Bearer TOKEN" --header "Content-Type: application/json"  http
://localhost:3000/gitlab-org/gitlab-test/prometheus/alerts/notify.json

# => 201 Created

HTTP Endpoint

This endpoints still returns information for created alerts :thumbsup:

  1. Setup HTTP Endpoint via Settings > Monitor > Alerts
  2. Fire an alert via curl
$ curl -v --request POST --data '{ "title": "Testing" }' --header "Authorization: Bearer TOKEN" --header "Content-Type: application/json"  http://localhost:3000/gitlab-org/gitlab-test/alerts/notify/test/ID.json
  1. Request succeeds and returns [{"iid":1,"title":"Testing"}] :white_check_mark:
  2. http://localhost:3000/gitlab-org/gitlab-test/-/alert_management/1/details is available

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Peter Leitzen

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Vitali Tatarintev marked this merge request as ready

    marked this merge request as ready

  • Vitali Tatarintev marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

    marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

  • Author Maintainer

    @sgarg_gitlab could you please review this one? Thanks

  • requested review from @sgarg_gitlab

  • Smriti Garg
  • Smriti Garg
  • Smriti Garg removed review request for @sgarg_gitlab

    removed review request for @sgarg_gitlab

  • requested review from @sgarg_gitlab

  • Smriti Garg approved this merge request

    approved this merge request

  • :wave: @sgarg_gitlab, thanks for approving this merge request.

    This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.

    For more info, please refer to the following links:

  • 🤖 GitLab Bot 🤖 added 1 deleted label

    added 1 deleted label

  • Vitali Tatarintev added 652 commits

    added 652 commits

    Compare with previous version

  • requested review from @splattael

  • Smriti Garg removed review request for @sgarg_gitlab

    removed review request for @sgarg_gitlab

  • Peter Leitzen changed the description

    changed the description

  • Peter Leitzen changed the description

    changed the description

  • Peter Leitzen changed the description

    changed the description

  • Peter Leitzen
  • Peter Leitzen approved this merge request

    approved this merge request

  • Peter Leitzen resolved all threads

    resolved all threads

  • Peter Leitzen enabled an automatic merge when the pipeline for 8d7fa6de succeeds

    enabled an automatic merge when the pipeline for 8d7fa6de succeeds

  • Vitali Tatarintev aborted the automatic merge because source branch was updated

    aborted the automatic merge because source branch was updated

  • Vitali Tatarintev added 844 commits

    added 844 commits

    Compare with previous version

  • Peter Leitzen enabled an automatic merge when the pipeline for cbad5b7b succeeds

    enabled an automatic merge when the pipeline for cbad5b7b succeeds

  • merged

  • Peter Leitzen mentioned in commit 8d88677b

    mentioned in commit 8d88677b

  • added workflowstaging label and removed workflowcanary label

  • mentioned in issue #372115 (closed)

  • Please register or sign in to reply
    Loading