Skip to content

Draft: POC - Download artifacts inside jobs via the API

David Dieulivol requested to merge 419910-run_rspec_jobs_before into master

Context

Closes #419910 (closed)

What does this MR do and why?

Instead of waiting for a few jobs to completely finish to start the RSpec jobs, we start the RSpec jobs, and block until we can retrieve artifacts from previous jobs.

This helps, because there might be a few minutes until we actually need the artifacts in those jobs.

Proof of work

See the trace for the MR pipeline:

  1. The RSpec jobs start right away:

Screenshot_2023-08-18_at_14.48.12

  1. They block until they can fetch the artifacts:

Screenshot_2023-08-18_at_14.50.45

  1. The pipeline is green 🎉

  2. Sidenote: Since I didn't change the RSpec as-if-foss jobs, you can see that they start after 3min30s (i.e. after the compile-test-assets job is finished):

Screenshot_2023-08-18_at_15.02.26

Private project

I also tested this change in a private project, and it now works as well there 🎉.

TL;DR: We cannot use JOB_CI_TOKEN for getting the job name (shell function job_name_to_job_id), as it's not a supported API path (we can download the artifacts with this token, but we cannot call pipelines/1234/:job_id 😢 ). I had to use PROJECT_TOKEN_FOR_CI_SCRIPTS_API_USAGE, which we already leverage in the detect-tests CI/CD job.

Performance improvements

Let's see how long it took until we could start Gitaly in the rspec system pg14 4/28 job:

Total waiting time until we could start gitaly: 2min15s + 1min22s = 3min37s

Compare this with a typical pipeline:

Total time until we could start gitaly: 3min35s + 0min22s + 2min15s = 6min12s

For that particular RSpec job, we saved 2min35s 🎉.

Note: compile-test-assets time greatly varies (from 2min up until 15min when no cache is available)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Dieulivol

Merge request reports