ci: Only keep relevant files after installing tests dependencies

Rémy Coutablerequested to merge
remove-gitaly-build-dir-after-compiling-it into master

What does this MR do?

Does it still work if Gitaly gems have changed compared to the gems in cache?

Not running gitaly-test-build anymore could lead to rspec frontend_fixture failing if gems were updated in Gitaly without being updated yet in the cache (which is only updated in the 2-hourly schedule), e.g. https://gitlab.com/gitlab-org/gitlab/-/jobs/916335823.

So I made sure gems are installed in gitaly-test-spawn so that gems that changed since the latest cached version are correctly installed, and I verified it by updating the gems in a Gitaly branch (gitaly@87db9065), updated the GITALY_SERVER_VERSION (!49014 (5c66406f)), and made sure the gems were properly installed (https://gitlab.com/gitlab-org/gitlab/-/jobs/918626328).

Conformity

Expected impact

We should save:

  • setup-test-env: 43 seconds, 148MB of artifacts

  • rspec *, run-dev-fixtures, run-dev-fixtures-ee, rspec frontend_fixture*, gitlab:setup, rspec fail-fast, rspec foss-impact: 94 seconds

  • graphql-reference-verify, memory-*, db:*, cache gems, update-tests-metadata: 20 seconds

  • 500 MR pipelines per day

  • 100 master pipelines per day

  • We have 135 rspec * jobs at max, so let's say in average we run 80 jobs (60 rspec system * jobs + some other rspec jobs).

  • graphql-reference-verify and db:* only run on specific changes so we'll assume it runs for 200 MRs per day

  • cache gems, update-tests-metadata only runs for master pipelines

A total of 600 pipelines per day means:

  • 600 * 43 = 25 800 seconds saved for setup-test-env
  • 600 * (80 + 6) * 94 = 4 850 400 seconds saved for rspec *, run-dev-fixtures, run-dev-fixtures-ee, rspec frontend_fixture*, gitlab:setup, rspec fail-fast, rspec foss-impact
  • (200 + 100) * 20 = 6 000 seconds saved for graphql-reference-verify, memory-*, db:*, cache gems, update-tests-metadata

We could save a total of 4 882 200 seconds per day, which is equivalent to 1 356 CPU minutes saved per day, as well as 1.5 minute on code pipelines due to rspec frontend_fixture* being faster!

Detailed analysis ### `setup-test-env`

Duration

Step Before After Diff
bundle exec ruby -I. -e 'require "config/environment"; TestEnv.init' 139 seconds 117 seconds -22 seconds
scripts/gitaly-test-build 148 seconds 145 seconds -3 seconds
Uploading artifacts... 35 seconds 17 seconds -18 seconds
Total 322 seconds 279 seconds -43 seconds

Artifacts

Item Before After Diff
Size compressed 269MB 121MB -148MB
Size uncompressed 549MB 257MB -292MB

rspec integration pg11 1/8

Step Before After Diff
Downloading artifacts 54 seconds 35 seconds -19 seconds
scripts/gitaly-test-build 75 seconds 0 seconds -75 seconds
Total 129 seconds 35 seconds -94 seconds

run-dev-fixtures

Step Before After Diff
Downloading artifacts 38 seconds 17 seconds -21 seconds
scripts/gitaly-test-build 75 seconds 0 seconds -75 seconds
Total 113 seconds 17 seconds -96 seconds

Other jobs where we save approximately 95 seconds:

  • run-dev-fixtures-ee
  • rspec frontend_fixture*
  • all the rspec* jobs
  • gitlab:setup
  • rspec fail-fast
  • rspec foss-impact

graphql-reference-verify

Step Before After Diff
Downloading artifacts 36 seconds 15 seconds -21 seconds
Total 36 seconds 15 seconds -21 seconds

Other jobs where we save approximately 20 seconds:

  • memory-*
  • db:*
  • cache gems
  • update-tests-metadata

Pre-merge tasks

Post-merge tasks

  • Post to #development
  • Post to the Engineering Week in Review

Related to #276217 (closed).

Edited by Rémy Coutable

Merge request reports