Skip to content

Generate failure signatures for individual RSpec tests

Important Note

We probably don't need this feature strictly speaking once Epic #28 (Build single test observability solution) is addressed. With that epic, we'll be able to identify test failures without a hash by using a "composite key" (filename + test name + error context), which might actually be a better approach than hashed signatures.

This issue remains useful for creating a uniform solution across all failure types (CI jobs + individual tests).

Problem Statement

RSpec test failures are currently categorized broadly but lack precise identification of individual failing tests. Teams cannot distinguish between different test failures or track specific flaky tests across pipeline runs.

Goal

Generate signatures for individual RSpec tests using test artifacts (JSON/XML reports, Allure reports), creating identifiers based on test file, test name, and normalized error context for precise failure tracking.

Input Data

Test data from test reports:

  • JSON files from RSpec output
  • Allure report data
  • JUnit XML reports

Output

Signature per test that can be added to the test data, likely including:

  • Filename
  • Test name
  • Hashed normalized failure context

Technical Considerations

Signature Components: May differ from CI job signatures - could include:

  • filename + test_name + hashed_normalized_failure_context
  • OR we could add the normalized failure context and compare those directly

Implementation Location: We might need to move signature generation logic from triage-ops to the gitlab_quality_test_tooling gem, since we push RSpec test data directly from CI jobs through that gem. We could then expand that data with failure signatures.

Related Work

  • Epic #28: Build single test observability solution (may make this optional)
  • Part of the broader failure signatures initiative
  • May require coordination with gitlab_quality_test_tooling gem
  • Builds on gitlab-org/quality/triage-ops!3654 (merged)
Edited by David Dieulivol