feat(alerts): Add prometheus query button to the Slack alert message

Introduction

In !3200 (merged) we changed the promql_template_1 attribute on alerts.

Previously it looked something like this:

      promql_template_1: >-
        gitlab_component_ops:rate_5m{environment="$environment", type="$type", stage="$stage", component="$component"}

This expression was never helpful, since the recording rule made is difficult to drill deeper into the problem.

In !3200 (merged), we switched to using fully-decomposed queries for the promql_template_1, additionally, the query uses an p95 histogram query instead of the apdex score, making it easier for operators to instantly grok. This allows for much faster drilling further into the data.

The above query was changed to, for example was changed to

      promql_template_1: |
        sum by (env,environment,tier,stage) (
          label_replace(rate(pgbouncer_stats_sql_transactions_pooled_total{type="patroni", tier="db", environment="{{ $labels.environment }}",stage="{{ $labels.stage }}"}[5m]), "_c", "0", "", "")
          or
          label_replace(rate(pgbouncer_stats_queries_pooled_total{type="patroni", tier="db", environment="{{ $labels.environment }}",stage="{{ $labels.stage }}"}[5m]), "_c", "1", "", "")
        )

Slack

This change adds an Action button to the AlertManager slack receiver to open the decomposed source data directly in the Thanos Query UI