Pipeline failure notification for AI Gateway

What does this MR do and why?

This is a corrective action for 2024-07-09: AI related E2E tests failing in sta... (gitlab-com/gl-infra/production#18252 - closed).

See Send a Slack message when a `main` pipeline fails (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#547) and feat: change vertex text model projects on gstg (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!1074).

See https://gitlab.slack.com/archives/CMA7DQJRX/p1720682308261169?thread_ts=1720600423.444219&cid=CMA7DQJRX

Expected impact & dry-runs

These are strongly recommended to assist reviewers and reduce the time to merge your change.

See https://gitlab.com/gitlab-org/quality/triage-ops/-/tree/master/doc/scheduled#testing-policies-with-a-dry-run on how to perform dry-runs for new policies.

See https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/doc/reactive/best_practices.md#use-the-sandbox-to-test-new-processors on how to make sure a new processor can be tested.

Action items

• If adding environment variables for reactive processors, update config/triage-web.yaml and .gitlab/ci/triage-web.yml
• (If applicable) Add documentation to the handbook pages for Triage Operations =>
• (If applicable) Identify the affected groups and how to communicate to them:
  • /cc @person_or_group =>
  • Relevant Slack channels =>
  • Engineering week-in-review