Add group similar results Slack alerting proof of concept
What does this MR do and why?
Relates to gitlab-org/quality/quality-engineering/team-tasks#3885
This MR implements automated Slack notifications for environment issues detected during CI test failures. When the gitlab_quality-test_tooling
gem's grouping functionality identifies patterns of similar failures (e.g., multiple HTTP 500 errors), it now exports this data and automatically posts alerts to the appropriate Slack channels.
Accompanying pipeline-common MR: gitlab-org/quality/pipeline-common!541
Key components:
-
Environment issue data export - Extends
relate-failure-issue
to export grouped failure data to JSON when environment issues are detected -
Slack notification enhancement - Extends
post-to-slack
to format and send environment issue alerts - CI integration - Integrates environment issue alerts into the existing pipeline-common QA reporting workflow
Benefits:
- Reduces noise by grouping similar failures into single alerts instead of individual test failure notifications
- Provides immediate visibility to environment-level issues affecting multiple tests
- Automatically posts to team Slack channels when environment issues are detected
- Maintains existing functionality when no environment issues are found
This is a proof of concept that provides immediate value while a more comprehensive solution is developed in gitlab-org/quality/quality-engineering&168
Screenshots or screen recordings
First test gathering existing 500 errors issue (used #e2e-run-fips mistakenly with its webhook in 1password ): https://gitlab.slack.com/archives/C07JMP82TDW/p1758828922240949):
The Slack message includes:
- Environment issue type (e.g., "HTTP 500 Internal Server Errors")
- Number of affected tests
- Error pattern details
- Summary statistics
- Link to comprehensive solution epic for future improvements
How to set up and validate locally
Test environment issue detection:
bundle exec relate-failure-issue --group-similar \ --input-files './spec/gitlab_quality/test_tooling/report/fixtures/rspec-reports/group-failures/job1/rspec-*.json' \ --project "gitlab-org/quality/e2e-test-issues" \ --token "<YOUR_TOKEN>" \ --dry-run
Expected output:
- Logs showing "Identified as environment error"
- "Found 1 groups after filtering"
- Creates
/tmp/environment_issues.json
with grouped failure data
Test Slack notification:
bundle exec post-to-slack
--environment-issues-file /tmp/environment_issues.json
--slack-webhook-url "$CI_SLACK_WEBHOOK_URL"
--channel "#your-test-channel"
--username "GitLab QA Bot"
--icon-emoji ""
Expected result:
- Slack message posted to specified channel
- Message includes environment issue details, affected test count, and summary
Test full CI integration:
The pipeline-common integration will automatically trigger when:
CREATE_TEST_FAILURE_ISSUES=true
QA_GROUP_SIMILAR_ENABLED=true
- Test failures are detected with similar patterns
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.