## Summary The Caproni shadow jobs in the `test-on-cng` child pipeline (`cng-instance-caproni`, `cng-relative-url-caproni`, `cng-registry-caproni`, `cng-oauth-caproni`, `cng-secrets-manager-caproni`) ignore E2E test selection. They always run their full scenario, even when `e2e-test-pipeline-generate` has cut the required jobs they shadow down to a few selected specs. They are `allow_failure: true`, but `e2e:test-on-cng` uses `trigger: strategy: depend`, so the parent MR pipeline still waits for them. In tier 2 MR pipelines with a selected spec list, they finish well after the required jobs and use about 10× the runner time. Once #623851 makes `cng-instance-caproni` blocking, every tier 2 MR with selected specs will gate on a full 10-shard suite. So this needs fixing before that flip. ## Cause - [`.caproni-test`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/test-on-cng/caproni.gitlab-ci.yml) sets the job-level variable `QA_TESTS: ""`. That overrides the top-level `QA_TESTS` written by [`PipelineCreator#variables_section`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/tools/ci/pipeline_creator.rb). - No Caproni job appears in `pipeline_mappings`, so `PipelineCreator#updated_job` never rescales their `parallel:` or sets `when: never` on them: - [`all.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/scenario/test/instance/all.rb): `%w[cng-instance cng-instance-ff-inverse]` - [`smoke.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/scenario/test/instance/smoke.rb): `%w[cng-relative-url]` - [`registry.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/scenario/test/integration/registry.rb): `%w[cng-registry]` - [`oauth.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/scenario/test/integration/oauth.rb): `%w[cng-oauth]` - [`secrets_manager.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/scenario/test/integration/secrets_manager.rb): `%w[cng-secrets-manager]` - `PipelineCreator` only edits job text that it finds in `.gitlab/ci/test-on-cng/main.gitlab-ci.yml`. The Caproni jobs live in `caproni.gitlab-ci.yml`, so mapping them as they are now would raise `Job definition not found`. !260217 hit exactly this when it tried to move `cng-instance-ff-inverse`, and kept that job in `main.gitlab-ci.yml` as a result. History: `QA_TESTS: ""` came in with the original smoke-only job (de33f7d4038b). 73de53c2ecee widened `cng-instance-caproni` to `Test::Instance::All` with `parallel: 10` for the #623164 comparison, but didn't add selection. ## Evidence The data comes from 60 successful tier 2 MR pipelines in the 24h before 2026-10-13T10:17Z ([collect job](https://gitlab.com/engwan/gitlab-pipeline-duration-agent/-/jobs/17136001661), full analysis in engwan/gitlab-pipeline-duration-agent#1). | Metric (tier 2) | Value | |---|---| | `e2e:test-on-cng` on the critical path | 34 of 60 (57%), p50 40m32s | | Slowest test job on the child critical path was a `*-caproni` job | 15 of 34 | | Time the Caproni jobs finished after the required test jobs (mean over the 34) | 116s | | Pipelines where selection cut the required jobs to 1–3 | 5 of 34, with Caproni finishing 219s, 497s, 606s, 629s and 1093s after the required jobs | | Test-stage runner minutes in those 5 pipelines, Caproni vs required | 212 vs 20 | Example: in [pipeline 2941520502](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2941520502), the required [`cng-instance`](https://gitlab.com/gitlab-org/gitlab/-/jobs/17133416431) ran as a single job on 12 selected spec files (12m20s). All 10 `cng-instance-caproni` shards still ran the full suite. The slowest, [17133416418](https://gitlab.com/gitlab-org/gitlab/-/jobs/17133416418), ran 47 examples and took 22m51s. ## Proposal Make the Caproni jobs follow the same selection as the jobs they shadow: 1. Remove `QA_TESTS: ""` from `.caproni-test`. 2. Add each Caproni job to the `pipeline_mappings` of its scenario, next to its twin (for example `%w[cng-instance cng-instance-ff-inverse cng-instance-caproni]`). 3. Move those job definitions from `caproni.gitlab-ci.yml` into `test-on-cng/main.gitlab-ci.yml`, as was done for `cng-instance-ff-inverse`. The other option is to teach `PipelineCreator` to read the included file. Removing the `QA_TESTS` override is safe for `cng-instance-ff-inverse`, which also extends `.caproni-test`. That job only runs when feature flags change, and in that case `ci.rake` always runs the full suite. Expected impact, as upper bounds: - Up to ~51s saved per tier 2 pipeline on average. - 3.6–18 minutes saved in each pipeline that has a selected spec list. - About 190 runner-minutes saved per such pipeline. ### Alternatives considered - **Run the Caproni jobs only on `master`.** This saves more time, but it conflicts with #623851, which needs MR-run comparison data. - **Move the Caproni jobs into a separate child pipeline without `strategy: depend`.** This adds another `build-cng-env` handoff, and I haven't checked whether the `build-cng` images and dotenv can be shared. ## Open questions - Was `QA_TESTS: ""` deliberate beyond the original smoke-only job? (@jay_mccure @mschoenlaub) - With this change, Caproni's MR pass-rate data covers only the selected specs, the same as the orchestrator jobs. Full-suite Caproni data would then come only from full-suite runs: `master`, framework changes, and `pipeline:run-all-e2e`. ## Related - #623851, which this blocks - &23118 - #623164 (closed) - !260217 - !261684 and #633324 (Caproni coverage mapping; they don't touch `QA_TESTS` or `pipeline_mappings`)