perf(e2e): parallelize deploy, gate diagnostics, prebake robot image
What does this MR do and why?
The e2e job is ~70% of main-pipeline wall-clock at a ~15-16 min median, and most of that is structural waiting: the GKG release queued behind the full GitLab install, three independent post-deploy steps running serially, a ~3-minute diagnostics dump on every green run, and the Robot pod installing its toolchain from the network each time. This MR removes that dead time without dropping any coverage, targeting a sub-10-minute green run.
Related Issues
Relates to #967 (closed)
Testing
Rendered the helmfile DAG and the robot-runner chart locally (with and without the image override), built the runner image locally, and validated the CI YAML. The e2e job on this MR is the real measurement — phase timings compared against the two-week trace baseline in #967 (closed).
Performance Analysis
Expected from measured phase medians: ~2.5 min from overlapping the GKG deploy with the GitLab boot, ~2.7 min from failure-only diagnostics, ~1 min from the prebaked runner image and single-wave suite pool, and ~2 min off MR runs from dropping DWARF (image shrinks ~5x). Failed-run burn halves via the tighter GitLab readiness timeout (healthy installs are ready in ~3 min; the old 20-minute deadline was the single most expensive failure mode).
- This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context — long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered
Baseline (11 main-success traces, 2026-06-19..07-03): provision 1.4m, setup.sh 6.0m, Robot 5.3m, after_script 2.9m, total ~15.8m. MR runs add ~4.5m (cargo dev build + 507 MB debug-image push).
Changes:
e2e/helmfile.yaml.gotmpl: gkg release drops itsneedson gitlab (secrets come from the bootstrap chart; no init container waits on Rails), so image pulls, indexer ClickHouse schema write, dispatcher, and health-check all overlap the GitLab boot. Note the gkg webserver/readyprobes Rails (project_info(1), 503 until reachable), so the release still completes only once GitLab is up — the win is overlap, not independence. GitLab release timeout 1200→600s: with the old serialized DAG, traces show gitlab fully Ready ~3 min after sync start; 29% of the window's failures were readiness timeouts burning the full 20-minute deadline.e2e/scripts/setup.sh:bootstrap-instance.sh(~60s Rails cold boot),patch-ch-dicts.sh(polls up to 300s), andpatch-ch-siphon-watermark.shnow run concurrently; they touch disjoint state. Considered and rejected: adding a table-existence poll to the watermark script — the gitlab release's readiness implies its migrations (including ClickHouse) completed beforehelmfile syncreturns, so the poll would guard an unreachable state..gitlab-ci.ymlafter_script +e2e/scripts/test.sh:dump-diagnostics.sh(walks every pod and ClickHouse table, ~2.7 min) now runs only whenCI_JOB_STATUS != success; test.sh keeps its own dump for non-CI runs. JUnit extraction and GKG pod logs still happen on every run in test.sh, so no report artifact is lost.e2e/Dockerfile.robot+e2e-robot-imagejob: prebaked python+robotframework/pabot image, content-addressed by the Dockerfile hash, built only when that file changes; a broken image build fails the pipeline. The runtime-install fallback covers the tag-missing case (local runs). Registry is public; the cluster pulls anonymously like it already does forgkg-e2e.e2e/charts/robot-runner: pabot processes 8→12 (one worker per downstream suite, single wave instead of two), pod 1 CPU/1Gi requests, 4 CPU/2Gi limits. Rate-limit safe: each suite polls with its own bot user (~20 req/min against the 60/min per-userorbit_querylimit).- MR image path:
CARGO_PROFILE_DEV_DEBUG=0drops DWARF from the dev build; backtraces keep function names via the symbol table. sccache takes a one-time cache-key miss. interruptible: true: superseded pipelines cancel and free cluster capacity; after_script (teardown) runs on cancel on current runners. The pre-existing leak classes (job timeout, runner death) are unchanged and belong to the namespace-reaper follow-up in #967 (closed).
Follow-ups tracked in #967 (closed): warm/persistent GitLab stack across runs (the path to a guaranteed sub-10 even on cold days), stale-namespace TTL reaper in the harness repo, cached CDC-config generation keyed on the gitlab/siphon pins.
Robot Framework bumped 7.2.2→7.4.2 (current latest; requests 0.9.7 and pabot 5.2.2 already latest); pip resolves the set cleanly and the e2e run on this MR exercises it.
%{all_commits}