Draft: POC - Download artifacts inside jobs via the API
Context
Closes #419910 (closed)
What does this MR do and why?
Instead of waiting for a few jobs to completely finish to start the RSpec jobs, we start the RSpec jobs, and block until we can retrieve artifacts from previous jobs.
This helps, because there might be a few minutes until we actually need the artifacts in those jobs.
Proof of work
See the trace for the MR pipeline:
- The RSpec jobs start right away:
-
Sidenote: Since I didn't change the RSpec as-if-foss jobs, you can see that they start after 3min30s (i.e. after the
compile-test-assets
job is finished):
Private project
I also tested this change in a private project, and it now works as well there
TL;DR: We cannot use JOB_CI_TOKEN
for getting the job name (shell function job_name_to_job_id
), as it's not a supported API path (we can download the artifacts with this token, but we cannot call pipelines/1234/:job_id
PROJECT_TOKEN_FOR_CI_SCRIPTS_API_USAGE
, which we already leverage in the detect-tests
CI/CD job.
Performance improvements
Let's see how long it took until we could start Gitaly in the rspec system pg14 4/28
job:
- Time until we start the job: 0s (see screenshot above)
- Time to setup the job until artifacts download: 18 + 27 + 14 + 14 + 62 = 2min15s (taken from https://gitlab.com/gitlab-org/gitlab/-/jobs/4899921581)
- Time to wait for/download the artifacts: 1min22s
Total waiting time until we could start gitaly: 2min15s + 1min22s = 3min37s
Compare this with a typical pipeline:
- Time until we start an RSpec job: Time it takes for compile-test-assets to finish -> 3min35s
- Time to download the artifacts: 22s
- Time to setup the job until we start gitaly: 18 + 27 + 14 + 14 + 62 = 135s (taken from https://gitlab.com/gitlab-org/gitlab/-/jobs/4899921581 as well)
Total time until we could start gitaly: 3min35s + 0min22s + 2min15s = 6min12s
For that particular RSpec job, we saved 2min35s
Note: compile-test-assets
time greatly varies (from 2min up until 15min when no cache is available)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.