<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/engineering/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=623851) </details> <!--IssueSummary end--> ## Problem Two jobs run the same test suite through different deployers. `cng-instance` is deployed by the orchestrator and is blocking today, because `.cng-test` sets no `allow_failure` so it defaults to false. `cng-instance-caproni` is deployed by Caproni and is non-blocking. Since https://gitlab.com/gitlab-org/gitlab/-/merge_requests/254865 landed on 2026-09-21, both run the full `Test::Instance::All` suite across 10 shards, so the duplication now costs a full suite deploy per pipeline for coverage already obtained. Since https://gitlab.com/gitlab-org/gitlab/-/merge_requests/262052 (merged 2026-10-15) the shadow jobs also honour tier-2 test selection, which removed the earlier blocker https://gitlab.com/gitlab-org/gitlab/-/work_items/633727. Being non-blocking also means nobody has to act on a Caproni failure, so the rig gets no real signal about whether it is trustworthy. ## Proposal Switch `cng-instance` itself to Caproni, in place and under its existing name, and delete the `cng-instance-caproni` shadow. This is the pattern already used for the two Caproni jobs that are blocking today: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/260217 changed `cng-instance-ff-inverse` from `extends: cng-instance` to `extends: .caproni-test` with `allow_failure: false`, and https://gitlab.com/gitlab-org/gitlab/-/merge_requests/261431 did the same for `cng-vue3-rollout`. In `.gitlab/ci/test-on-cng/main.gitlab-ci.yml`: - [ ] `cng-instance` becomes `extends: .caproni-test` with `allow_failure: false` (the template is non-blocking by default), `parallel: 10`, and the variables the shadow carries today: `QA_RUN_IN_PARALLEL: "true"`, `QA_PARALLEL_PROCESSES: 4` and `E2E_CODE_PATHS_MAPPING: "true"` - [ ] `cng-instance-caproni` is deleted, and `export-code-paths-mapping-caproni` drops it from `needs` (the registry and secrets-manager shadows still feed that job) - [ ] The definition stays in `main.gitlab-ci.yml`. `QA::Tools::Ci::PipelineCreator` reads that file as raw text and does not follow `include:`, so moving the job elsewhere makes the generator abort with "Job definition not found" ## Sequencing The parent epic at https://gitlab.com/groups/gitlab-org/-/work_items/23118 requires `test-on-cng` to stay runnable as a fallback for one milestone. After the switch that fallback is a revert of the switching MR. `.cng-test` and `.cng-test-parallel` stay in the file because `cng-registry` and `cng-secrets-manager` still use them, so a revert needs nothing re-added. Before the switch: 1. The Phase 2 written recommendation, the decision gate on https://gitlab.com/groups/gitlab-org/-/work_items/23117 2. A repeat of the multi-day MR-pipeline comparison from the Phase 3 epic (https://gitlab.com/groups/gitlab-org/-/work_items/23118#note_3960133782), after the sign-in readiness gate merged in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/259905 on 2026-10-08. The 2026-09-27 to 2026-10-06 sample (5 of 70 Caproni pipelines failed, 4 of them deploy readiness races, orchestrator 0 of 70) predates that fix. The only post-fix data is a one-day scheduled sample with 1 failed shard in 180, a CNPG CRD readiness race during `caproni up` 3. Either a deploy retry equivalent to the orchestrator's `--retry 1`, or a fix for the remaining `caproni up` readiness races 4. Diagnostics fit for a blocking job, see below ## Watch out for 1. Documentation. `cng-instance` is named in `doc/development/testing_guide/end_to_end/feature_flag_testing.md`, `doc/development/testing_guide/end_to_end/best_practices/feature_flags.md`, `doc/development/testing_guide/end_to_end/debugging_end_to_end_test_failures.md` and `doc/development/testing_guide/end_to_end/test_pipelines.md`. Nothing to change, the name is kept. The same holds for the `pipeline_mappings` in `qa/qa/scenario/test/instance/all.rb` and the knapsack runtime data keyed on the job name. 1. Coverband. The orchestrator job passes `COVERBAND_ENABLED` through `.with-coverband-arg`, while the Caproni rig enables it only when `E2E_CODE_PATHS_MAPPING` is `"true"`. A per-test comparison on 2026-10-22 showed the Caproni code paths mapping matches the orchestrator's within the orchestrator's own run-to-run variance: https://gitlab.com/gitlab-org/gitlab/-/work_items/633324#note_4068680310. The mapping now flows through `export-code-paths-mapping` under `master/e2e-test-on-cng`, the prefix predictive test selection reads. 1. Failure diagnosis. `qa/caproni/scripts/save-cluster-logs.sh` only reads the `gitlab` namespace, so CloudNativePG, Valkey and Garage logs are not captured, and the `cng-oauth-caproni` bundle is empty until https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264000 lands. That is tolerable for a job nobody has to triage and not tolerable for a blocking one. 1. Reliability evidence. The only multi-day sample predates the sign-in readiness gate, so it has to be re-measured after that fix, as described under Sequencing. 1. `cng-relative-url` extends `cng-instance`, so switching the instance job to `.caproni-test` moves the relative-url job onto Caproni implicitly, and its orchestrator-only `EXTRA_DEPLOY_VALUES` override (`--set global.appConfig.relativeUrlRoot=...`) would then do nothing. The switching MR has to convert `cng-relative-url` in the same change to the shape its shadow `cng-relative-url-caproni` already uses (`extends: .caproni-test`, `QA_SCENARIO: Test::Instance::Smoke`, `QA_RUN_IN_PARALLEL: "true"`, `QA_PARALLEL_PROCESSES: 4`, `QA_RELATIVE_URL_ROOT: "/relative"`, `allow_failure: false`), delete that shadow, and drop its name from `pipeline_mappings` in `qa/qa/scenario/test/instance/smoke.rb`. The alternative is to re-point it at `.cng-test-parallel` until its own switch. ## Exit criteria - [ ] `cng-instance` deploys through Caproni, under its existing name, with `allow_failure: false`, and `cng-instance-caproni` no longer exists - [ ] Predictive test selection reads a Caproni-produced mapping from `master/e2e-test-on-cng` without a drop in selected tests - [ ] A milestone of green blocking runs has elapsed without reverting to the orchestrator - [ ] The orchestrator templates are removed once no `cng-` job uses them (this last step may be its own issue)