Skip to content

Investigate better long term test case strategy

Context

Our e2e test reporting system works as follows:

  • Test Cases: One permanent issue per e2e spec that serves as a central hub. Linked in the test code itself. Contains a list of results issues to preserve test result history.
  • Results Issues: Temporary issue per e2e spec that collects test results in the form of a comment with stacktrace for every failure. Automatically closed after 500 notes to prevent timeouts.

When tests run with reporting enabled:

  • The reporting script updates both the test case and results issue for each spec

Test cases are updated with:

  • Pass/fail environment labels (only when status changes)
  • Any new results issue link in the description

Results Issues are updated with:

  • Pass/fail environment labels (only when status changes)
  • A new comment for every failure containing the stacktrace (similar stacktraces are grouped in threads)

The design intent was for test cases to be permanent reference points that maintain a complete history of all corresponding results issues, which get recreated periodically. This avoids needing to update spec code with new issue links whenever a results issue fills up.

Problem

Test Cases are now timing out during updates (example failure, example test case).

Root cause: While test cases don't display notes/comments in the UI, GitLab still creates internal notes every time we update labels or descriptions. Even though we only update the description when a new results issue is created and only update labels when pass/fail status changes, older test cases have accumulated thousands of these hidden notes, causing API timeouts when the reporting script tries to update them. There is also the triage bot that updates the description with contributor links, which get removed when the description is updated, then added back again.

This same problem affects long standing failure and slow test issues: gitlab-org/ruby/gems/gitlab_quality-test_tooling#113 (comment 2703336189)

Temporary solutions

  • Do not add contributor links on test case type issues.
  • Archive oldest test cases (the scripts will create new test cases automatically if the testcase linked is archived and it can't find a new one on searching) and update newly recreated test cases with the historic results issue links from the old, also update the testcase links in e2e specs with the new test case url
Edited by Désirée Chevalier