Generate failure signatures for individual RSpec tests
Important Note
We probably don't need this feature strictly speaking once Epic #28 (Build single test observability solution) is addressed. With that epic, we'll be able to identify test failures without a hash by using a "composite key" (filename + test name + error context), which might actually be a better approach than hashed signatures.
This issue remains useful for creating a uniform solution across all failure types (CI jobs + individual tests).
Problem Statement
RSpec test failures are currently categorized broadly but lack precise identification of individual failing tests. Teams cannot distinguish between different test failures or track specific flaky tests across pipeline runs.
Goal
Generate signatures for individual RSpec tests using test artifacts (JSON/XML reports, Allure reports), creating identifiers based on test file, test name, and normalized error context for precise failure tracking.
Input Data
Test data from test reports:
- JSON files from RSpec output
- Allure report data
- JUnit XML reports
Output
Signature per test that can be added to the test data, likely including:
- Filename
- Test name
- Hashed normalized failure context
Technical Considerations
Signature Components: May differ from CI job signatures - could include:
-
filename
+test_name
+hashed_normalized_failure_context
- OR we could add the normalized failure context and compare those directly
Implementation Location: We might need to move signature generation logic from triage-ops to the gitlab_quality_test_tooling
gem, since we push RSpec test data directly from CI jobs through that gem. We could then expand that data with failure signatures.
Related Work
- Epic #28: Build single test observability solution (may make this optional)
- Part of the broader failure signatures initiative
- May require coordination with
gitlab_quality_test_tooling
gem - Builds on gitlab-org/quality/triage-ops!3654 (merged)