Skip to content

Add group similar results Slack alerting proof of concept

What does this MR do and why?

Relates to gitlab-org/quality/quality-engineering/team-tasks#3885

This MR implements automated Slack notifications for environment issues detected during CI test failures. When the gitlab_quality-test_tooling gem's grouping functionality identifies patterns of similar failures (e.g., multiple HTTP 500 errors), it now exports this data and automatically posts alerts to the appropriate Slack channels.

Accompanying pipeline-common MR: gitlab-org/quality/pipeline-common!541

Key components:

  1. Environment issue data export - Extends relate-failure-issue to export grouped failure data to JSON when environment issues are detected
  2. Slack notification enhancement - Extends post-to-slack to format and send environment issue alerts
  3. CI integration - Integrates environment issue alerts into the existing pipeline-common QA reporting workflow

Benefits:

  • Reduces noise by grouping similar failures into single alerts instead of individual test failure notifications
  • Provides immediate visibility to environment-level issues affecting multiple tests
  • Automatically posts to team Slack channels when environment issues are detected
  • Maintains existing functionality when no environment issues are found

This is a proof of concept that provides immediate value while a more comprehensive solution is developed in gitlab-org/quality/quality-engineering&168

Screenshots or screen recordings

First test gathering existing 500 errors issue (used #e2e-run-fips mistakenly with its webhook in 1password ): https://gitlab.slack.com/archives/C07JMP82TDW/p1758828922240949):

test slack message.png

The Slack message includes:

  • Environment issue type (e.g., "HTTP 500 Internal Server Errors")
  • Number of affected tests
  • Error pattern details
  • Summary statistics
  • Link to comprehensive solution epic for future improvements

How to set up and validate locally

Test environment issue detection:

bundle exec relate-failure-issue --group-similar \ --input-files './spec/gitlab_quality/test_tooling/report/fixtures/rspec-reports/group-failures/job1/rspec-*.json' \ --project "gitlab-org/quality/e2e-test-issues" \ --token "<YOUR_TOKEN>" \ --dry-run

Expected output:

  • Logs showing "Identified as environment error"
  • "Found 1 groups after filtering"
  • Creates /tmp/environment_issues.json with grouped failure data

Test Slack notification:

bundle exec post-to-slack 
--environment-issues-file /tmp/environment_issues.json 
--slack-webhook-url "$CI_SLACK_WEBHOOK_URL" 
--channel "#your-test-channel" 
--username "GitLab QA Bot" 
--icon-emoji ""

Expected result:

  • Slack message posted to specified channel
  • Message includes environment issue details, affected test count, and summary

Test full CI integration:

The pipeline-common integration will automatically trigger when:

  • CREATE_TEST_FAILURE_ISSUES=true
  • QA_GROUP_SIMILAR_ENABLED=true
  • Test failures are detected with similar patterns

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Tim Beauchamp

Merge request reports

Loading