WIP: Test rspec impact `!41081` on backend changes

What does this MR do?

Compare failed pipeline on backend change !41258 with and without rspec impact.

Without rspec impact

• pipeline https://gitlab.com/gitlab-org/gitlab/-/pipelines/185144017:
  • pipeline failed after 00:45:13
  • duration of test jobs until completed (either passed/failed):
    • rspec unit - 20 minutes 35 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717454138
    • rspec integration - 22 minutes 22 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717454205
    • rspec system - 16 minutes 55 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717454220
    • rspec-ee unit - 18 minutes 28 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717454313
    • rspec-ee integration - 14 minutes 29 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717454313
    • rspec-ee system - 16 minutes 11 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717454325
    • frontend-fixtures: 16 minutes 31 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717609024
    • jest: 16 minutes 49 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717609060
    • jest-integration: 5 minutes 23 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717609072
    • karma: 5 minutes 17 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717609056

With rspec impact

• pipeline https://gitlab.com/gitlab-org/gitlab/-/pipelines/185176184
  • pipeline failed & canceled after 00:27:47
  • rspec impact failed after 12 minutes 56 seconds: https://gitlab.com/gitlab-org/gitlab/-/jobs/717607788
  • duration of test jobs until cancelled:
    • rspec unit - 14 minutes 57 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607314
    • rspec integration - 14 minutes 54 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607380
    • rspec system - 14 minutes 52 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607398
    • rspec-ee unit - 14 minutes 42 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607604
    • rspec-ee integration - 14 minutes 40 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607622
    • rspec-ee system - 14 minutes 40 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607631
    • frontend-fixtures: 14 minutes 59 seconds - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607217
    • jest: 0 minutes - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607251
    • jest-integration: 0 minutes - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607263
    • karma: 0 minutes - https://gitlab.com/gitlab-org/gitlab/-/jobs/717607244

Approximate runner time saved in this example, taking into account parallel job counts [formula: parallel count * (full job duration - cancelled job duration)]:

• rspec unit: 20 * (20:35 - 14:57) ~ approx 110 minutes
• rspec integration: 8 * (22:22 - 14:54) ~ approx 56 minutes
• rspec system: 24 * (16:55 - 14:52) ~ approx 48 minutes
• rspec-ee unit: 10 * (18:28 - 14:42) ~ approx 35 minutes
• rspec-ee integration: 4 * (14:29 - 14:54) ~ approx 0 minutes
• rspec-ee system: 6 * (16:11 - 14:40) ~ approx 6 minutes
• frontend-fixtures: 1 * (16:31 - 14:59) ~ approx 1 minute
• jest: 4 * (16:49 - 00:00) ~ approx 67 minutes
• jest-integration: 1 * (5:23 - 00:00) ~ approx 5 minutes
• karma: 1 * (5:17 - 00:00) ~ approx 5 minutes

Total approx runner time savings: 333 minutes

Caveat: This analysis excludes additional potential savings from geo, as-if-foss, pg 12 jobs and other miscellaneous jobs such as coverage, metadata.

Summary findings

For a 11-file change backend MR:

• Total wall time savings: approx 18 minutes
• Total runner time savings: approx 333 minutes

Overall runner time saved is a function of:

1. Duration of rspec-impact job: The quicker rspec-impact job completes (and fails), the earlier the pipeline and other jobs are cancelled
   • This in turn is a function of number of backend files changed in the MR
2. Number of parallelism of various jobs. This has 2-sided effect:
   • The higher the parallel count, the higher the saving due to multiplication effect. BUT
   • The more parallel count we have, the quicker each individual job is, and might complete before rspec-impact, negating the effect of the short circuit.

Notes:

number of parallel tests:

.rspec-unit-parallel:
  parallel: 20
.rspec-ee-unit-parallel:
  parallel: 10
.rspec-integration-parallel:
  parallel: 8
.rspec-ee-integration-parallel:
  parallel: 4
.rspec-system-parallel:
  parallel: 24
.rspec-ee-system-parallel:
  parallel: 6
jest:
  parallel: 4