Full query is not listed in incident summary for non-gitlab configured prometheus alerts
Summary
When a gitlab instance has incident management enabled, a triggered prometheus alert should cause an incident issue to be created with the details of the alert.
One of the fields we expect to see for prometheus alerts in the "Summary" section of the issue description is full_query. For gitlab-configured alerts, this seems to be working correctly. There is fallback logic in place, however, which does not handle the likely format of the alert from prometheus correctly.
For example, this is the body of an incident created using https://gitlab.com/splattael/gitlab-manual-prometheus:
New: Incident 8
Summary
Start time: 2020-01-07T19:44:17+00:00
Alert Details
receiver: gitlab
status: firing
alerts: [{"status"=>"firing", "labels"=>{"alertname"=>"AlertmanagerNotificationsFailing3", "pager"=>"pagerduty", "service"=>"alertmanager", "severity"=>"s1"}, "annotations"=>{"description"=>"Alertmanager is seeing errors for integration ", "runbook"=>"troubleshooting/alertmanager-notification-failures.md", "title"=>"Alertmanager is failing sending notications"}, "startsAt"=>"2020-01-07T19:13:41.421665749Z", "endsAt"=>"0001-01-01T00:00:00Z", "generatorURL"=>"http://184dd1f1425f:9090/graph?g0.expr=vector%281%29&g0.tab=1"}]
groupLabels.alertname: AlertmanagerNotificationsFailing3
commonLabels.severity: s1
commonLabels.service: alertmanager
commonLabels.pager: pagerduty
commonLabels.alertname: AlertmanagerNotificationsFailing3
commonAnnotations.title: Alertmanager is failing sending notications
commonAnnotations.runbook: troubleshooting/alertmanager-notification-failures.md
commonAnnotations.description: Alertmanager is seeing errors for integration
externalURL: http://f4a318fd933b:9093
version: 4
groupKey: {}:{alertname="AlertmanagerNotificationsFailing3"}The logic in https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/alerting/alert.rb#L152 looks for the query in the generatorUrl as a top-level key, but prometheus may have the generatorUrl nested under the alerts key. So the query is available, but we are not presenting it to the user.
Steps to reproduce
In a project with incident management enabled, configure the alerts endpoint to point to a running prometheus instance created from https://gitlab.com/splattael/gitlab-manual-prometheus. An issue will appear which is formatted comparably to above.
What is the current bug behavior?
The full_query key does not appear under the Summary section of an incident issue for self-managed prometheus installs.
What is the expected correct behavior?
The full_query key appears based on the generatorUrl key of the triggered alerts.
Possible fixes
I'm guessing this was configured & tested with a setup which has the generatorUrl as a top-level key, so we should probably add a check in https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/alerting/alert.rb#L152 which iterates over the alerts array if it's not present.