Fetch only the latest commit in e2e-test-pipeline-generate
What does this MR do?
Sets GIT_DEPTH: "1" for e2e-test-pipeline-generate. It's the slowest part of the job: with the default depth of 20 it fetches about 325,000 objects (269 MiB), and get_sources takes about 90 seconds. Jobs with depth 1 fetch about 136,000 objects in 26–33 seconds.
The job doesn't need Git history. It gets the MR's changed files from the API (mr_diff in qa/tasks/helpers/util.rb calls merge_requests/:iid/changes) and only reads files in the checkout. e2e-test-pipeline-generate as-if-foss extends this job, so it gets the same change.
Measurements and effect on wall clock
From https://gitlab.com/gitlab-org/gitlab/-/pipelines/2957945860:
| Job | Git depth | Objects fetched | get_sources |
|---|---|---|---|
e2e-test-pipeline-generate |
20 | 324,999 (269 MiB) | 92s |
compile-production-assets |
1 | 136,344 (201 MiB) | 26s |
cache:assets-hash |
1 | 136,344 (201 MiB) | 33s |
- Today
e2e:test-on-cngalso waits for the asset jobs, so this saves runner time but not wall clock. - With !263017 (merged),
e2e:test-on-cngonly waits for this job andcache:assets-hash. Then this job decides when E2E starts, and E2E would start about a minute earlier in pipelines where the assets image already exists.
How to verify
In this MR's pipeline, e2e-test-pipeline-generate logs Fetching changes with git depth set to 1..., get_sources takes around 30 seconds, and the job still generates the E2E pipeline files.
Related issues
Checklist
Pre-merge
Consider the effect of the changes in this merge request on the following:
- Different pipeline types: only the checkout depth of
e2e-test-pipeline-generate(and its as-if-foss variant) changes, in every pipeline type that runs it. The job gets its diff from the API, not from Git history. - Non-canonical projects:
-
gitlab-foss: same as above. -
security: same as above. -
dev: same as above. - personal forks: same as above. Without the API token the job runs the full test suite, as before.
-
- Pipeline performance: about a minute less per run of this job.
If new jobs are added:
- Change-related rules (e.g. frontend/backend/database file changes): N/A, no new jobs
- Frequency they are running (MRs, main branch, nightly, bi-hourly): N/A, no new jobs
- Add a duration chart to https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations if there are new jobs added to merge request pipelines: N/A, no new jobs
This will help keep track of expected pipeline time and cost increases.
Post-merge
- Consider communicating these changes to the broader team following the communication guideline for pipeline changes