Skip to content

Treat test failures as identical if the error message is the same even if the stack trace is slightly different

After https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/438 identical test failures are reported as brief comments in a discussion instead of as duplicate comments.

However, failures are matched on error message and stack trace, and they have to be identical to match. This is a problem when some part of the content varies with the test, e.g., if a user is created with a random username and the username is included in the stack trace. For example, gitlab-org/quality/testcases#722 (comment 316557811)

QA::Resource::ApiFabricator::ResourceNotFoundError:
  Expected one user with username qa-user-0d71a906e469dd34 but found: `[]`.
  ./qa/resource/user.rb:139:in `fetch_id'
QA::Resource::ApiFabricator::ResourceNotFoundError:
  Expected one user with username qa-user-83355bf122809aac but found: `[]`.
  ./qa/resource/user.rb:139:in `fetch_id'

Or when a line number changes.

In that case, every failure is considered different because it has a different username.

Proposal

Match exactly on error message, but allow stack traces to differ a little (maybe 95% match? Maybe using levenshtein distance)

We can also do something like:

require 'diffy'
puts Diffy::Diff.new(text1, text2, context: 0)
-Shared Example Group: "user access" called from ./qa/specs/features/ee/browser_ui/1_manage/group/group_saml_enforced_sso_spec.rb:97
+Shared Example Group: "user access" called from ./qa/specs/features/ee/browser_ui/1_manage/group/group_saml_enforced_sso_spec.rb:100
Edited by Mark Lapierre