Fetch only the latest commit in e2e-test-pipeline-generate

What does this MR do?

Sets GIT_DEPTH: "1" for e2e-test-pipeline-generate. It's the slowest part of the job: with the default depth of 20 it fetches about 325,000 objects (269 MiB), and get_sources takes about 90 seconds. Jobs with depth 1 fetch about 136,000 objects in 26–33 seconds.

The job doesn't need Git history. It gets the MR's changed files from the API (mr_diff in qa/tasks/helpers/util.rb calls merge_requests/:iid/changes) and only reads files in the checkout. e2e-test-pipeline-generate as-if-foss extends this job, so it gets the same change.

Measurements and effect on wall clock

From https://gitlab.com/gitlab-org/gitlab/-/pipelines/2957945860:

Job Git depth Objects fetched get_sources
e2e-test-pipeline-generate 20 324,999 (269 MiB) 92s
compile-production-assets 1 136,344 (201 MiB) 26s
cache:assets-hash 1 136,344 (201 MiB) 33s
  • Today e2e:test-on-cng also waits for the asset jobs, so this saves runner time but not wall clock.
  • With !263017 (merged), e2e:test-on-cng only waits for this job and cache:assets-hash. Then this job decides when E2E starts, and E2E would start about a minute earlier in pipelines where the assets image already exists.

How to verify

In this MR's pipeline, e2e-test-pipeline-generate logs Fetching changes with git depth set to 1..., get_sources takes around 30 seconds, and the job still generates the E2E pipeline files.

Checklist

Pre-merge

Consider the effect of the changes in this merge request on the following:

  • Different pipeline types: only the checkout depth of e2e-test-pipeline-generate (and its as-if-foss variant) changes, in every pipeline type that runs it. The job gets its diff from the API, not from Git history.
  • Non-canonical projects:
    • gitlab-foss: same as above.
    • security: same as above.
    • dev: same as above.
    • personal forks: same as above. Without the API token the job runs the full test suite, as before.
  • Pipeline performance: about a minute less per run of this job.

If new jobs are added:

This will help keep track of expected pipeline time and cost increases.

Post-merge

Edited by Heinrich Lee Yu

Merge request reports

Loading
Loading