## Goal Consolidate and align on the test reporting tools and processes between the Rails test suite (https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec & https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/spec) and the E2E test suite (https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/specs). ## Values alignment By using consolidated test reporting tools, we'll: - [Collaboration] allow more collaboration between Quality Engineering (DRI of `gitlab-qa`, where the test report tooling were originally implemented) and Engineering Productivity (as well as Development since they're the ultimate consumers of test reports, e.g. flaky test issues are assigned to DevOps groups). - [Results] get consistent test reporting. We'll also grow the potential owners of the test report tooling (instead of just being Quality Engineering). - [Efficiency] be more efficient by reducing duplication & maintenance costs. - [Transparency] use a SSOT for all the test reporting tools. ## Background and specificities Documentation: - https://handbook.gitlab.com/handbook/engineering/quality/#test-results-tracking/ - https://docs.gitlab.com/ee/development/testing_guide/test_results_tracking.html ### Background I originally implemented a tool to create issues automatically for failing tests (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104983), and then realized that Quality Engineering already have a very similar tool in the `gitlab-qa` project. I took a step back and saw an opportunity to optimize globally by consolidating our test reporting tools. ### Test cases, statuses, and historical failures - https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases is the SSOT for individual test cases, which are referenced in test with the testcase: metadata - https://gitlab.com/gitlab-org/quality/testcases/-/issues is the SSOT for test results: 1 issue per test with labels specifying the latest result per environment. This is used to keep historical data on successes / failures. From Mark L: > I use it every time I'm on pipeline triage. When there's a failure that's been happening for a while and I want to find out when it started. Or if I want an idea of how flaky a test is and it's been failing for different reasons. - https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=QA&first_page_size=100 is the SSOT for test failures on which groups can work to resolve - https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues is the SSOT for test sessions, which link to https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases and https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=QA&first_page_size=100 - [Rémy](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/issues/209#note_1368041558): > What exactly are you missing from the test reports feature? Are you thinking about replacing https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues somehow? - [Mark](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/issues/209#note_1371310840): > the main one would be a history of test results: https://gitlab.com/groups/gitlab-org/-/epics/3129 > And once we're storing test data we should be able to improve failure reporting and management. For example, automatically linking failures for the same test across pipelines and projects. > We can already see if a failure is new *[within a project](https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html#how-it-works)*[ (target branch vs source branch)](https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html#how-it-works), but it would be helpful if we could compare with pipelines in other projects too. That could make it much easier to know where failures are occuring. For example, if a test fails in master, a report could show if (and when) the test failed in any other pipelines, including MRs. If we could do all of that I think we could create a replacement for testcase sessions that could be a lot better. I think we should try to consolidate this and use a similar setup for Rails tests. We don't really need https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues nor https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases (we'd use "boring" test identifier, i.e. `<filename>-<test_description>` instead of formal test cases). Currently, the experiment at https://gitlab.com/gitlab-org/quality/engineering-productivity/flaky-tests-playground/-/issues is roughly similar to https://gitlab.com/gitlab-org/quality/testcases/-/issues since we update the issues' description with the list of failures. I think we should be able to combine https://gitlab.com/gitlab-org/quality/testcases/-/issues and https://gitlab.com/gitlab-org/gitlab/-/issues by keeping the list of failures in the issue description, while allowing teams to work on the failure and resolve it. ### Tooling We currently have some duplication and differences in the way we manage tests reporting and test tooling in general between Rails (`spec/`) and E2E test suites (`qa/qa/specs`). - RSpec formatters: we have various formatters at https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/support/formatters, including one for quarantined tests, and we're going to have the same need for https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/issues/204. - Knapsack tooling: to download/upload Knapsack reports - Rails: https://gitlab.com/gitlab-org/gitlab/-/blob/c080cf310c8c17afce53e79adb63f99b343add49/scripts/rspec_helpers.sh#L6-9 / https://gitlab.com/gitlab-org/gitlab/-/blob/c080cf310c8c17afce53e79adb63f99b343add49/scripts/rspec_helpers.sh#L23 and the final Pages report at https://gitlab-org.gitlab.io/gitlab/knapsack/report-master.json. - E2E: https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/support/knapsack_report.rb - We also have https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/helpers/wait_for_requests.rb and https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/support/wait_for_requests.rb. - We also have https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/helpers/stub_env.rb and https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/support/helpers/stub_env.rb. - Lastly we also have some tooling in `gitlab-org/gitlab-qa` that we could reuse for [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab/-/issues/361666), [`gitlab-org/gitaly`](https://gitlab.com/gitlab-org/gitaly/-/issues/4788), [`gitlab-org/customers-gitlab-com`](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6228), etc.: https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/lib/gitlab/qa/reporter.rb ## Solution It would be great to consolidate the Rails/E2E tooling into Rails, and make sure we reuse the files in the E2E Docker image. ### Iteration 1 - [x] Create a `gitlab_quality-test_tooling` gem to basically encapsulate `gitlab-qa-report`. The `gitlab_quality` "prefix" would be to state that this is a gem used by the Quality sub-department of GitLab. If we name it only `gitlab-test_tooling`, that could be misleading as an external-targeted gem. Issue: https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/issues/209 ### Iteration 2 - [x] Generalize `relate-failure-issue` so that it can be used for other use-cases than just E2E tests ([`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab/-/issues/361666), [`gitlab-org/gitaly`](https://gitlab.com/gitlab-org/gitaly/-/issues/4788), [`gitlab-org/customers-gitlab-com`](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6228), etc.). Issue: https://gitlab.com/gitlab-org/ruby/gems/gitlab_quality-test_tooling/-/issues/7 / gitlab-org/ruby/gems/gitlab_quality-test_tooling!22 ### Iteration 3 - [x] Use the generalized script to automatically create Rails test failure issues (similarly to how we already do it for E2E test failures): https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122008 / https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124208 #### Iteration 4 - [ ] Move [duplicated RSpec support files](https://gitlab.com/gitlab-org/gitlab/-/issues/458885) to `gems/gitlab-rspec` and use them in both Rails and `qa/` ### Future iterations Move shared RSpec logic to `gitlab-rspec` (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121676) so that it's easier to re-use in `qa/` image.