perf(e2e): parallelize deploy, gate diagnostics, prebake robot image

What does this MR do and why?

The e2e job is ~70% of main-pipeline wall-clock at a ~15-16 min median, and most of that is structural waiting: the GKG release queued behind the full GitLab install, three independent post-deploy steps running serially, a ~3-minute diagnostics dump on every green run, and the Robot pod installing its toolchain from the network each time. This MR removes that dead time without dropping any coverage, targeting a sub-10-minute green run.

Relates to #967 (closed)

Testing

Rendered the helmfile DAG and the robot-runner chart locally (with and without the image override), built the runner image locally, and validated the CI YAML. The e2e job on this MR is the real measurement — phase timings compared against the two-week trace baseline in #967 (closed).

Performance Analysis

Expected from measured phase medians: ~2.5 min from overlapping the GKG deploy with the GitLab boot, ~2.7 min from failure-only diagnostics, ~1 min from the prebaked runner image and single-wave suite pool, and ~2 min off MR runs from dropping DWARF (image shrinks ~5x). Failed-run burn halves via the tighter GitLab readiness timeout (healthy installs are ready in ~3 min; the old 20-minute deadline was the single most expensive failure mode).

  • This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context — long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered

Baseline (11 main-success traces, 2026-06-19..07-03): provision 1.4m, setup.sh 6.0m, Robot 5.3m, after_script 2.9m, total ~15.8m. MR runs add ~4.5m (cargo dev build + 507 MB debug-image push).

Changes:

  • e2e/helmfile.yaml.gotmpl: gkg release drops its needs on gitlab (secrets come from the bootstrap chart; no init container waits on Rails), so image pulls, indexer ClickHouse schema write, dispatcher, and health-check all overlap the GitLab boot. Note the gkg webserver /ready probes Rails (project_info(1), 503 until reachable), so the release still completes only once GitLab is up — the win is overlap, not independence. GitLab release timeout 1200→600s: with the old serialized DAG, traces show gitlab fully Ready ~3 min after sync start; 29% of the window's failures were readiness timeouts burning the full 20-minute deadline.
  • e2e/scripts/setup.sh: bootstrap-instance.sh (~60s Rails cold boot), patch-ch-dicts.sh (polls up to 300s), and patch-ch-siphon-watermark.sh now run concurrently; they touch disjoint state. Considered and rejected: adding a table-existence poll to the watermark script — the gitlab release's readiness implies its migrations (including ClickHouse) completed before helmfile sync returns, so the poll would guard an unreachable state.
  • .gitlab-ci.yml after_script + e2e/scripts/test.sh: dump-diagnostics.sh (walks every pod and ClickHouse table, ~2.7 min) now runs only when CI_JOB_STATUS != success; test.sh keeps its own dump for non-CI runs. JUnit extraction and GKG pod logs still happen on every run in test.sh, so no report artifact is lost.
  • e2e/Dockerfile.robot + e2e-robot-image job: prebaked python+robotframework/pabot image, content-addressed by the Dockerfile hash, built only when that file changes; a broken image build fails the pipeline. The runtime-install fallback covers the tag-missing case (local runs). Registry is public; the cluster pulls anonymously like it already does for gkg-e2e.
  • e2e/charts/robot-runner: pabot processes 8→12 (one worker per downstream suite, single wave instead of two), pod 1 CPU/1Gi requests, 4 CPU/2Gi limits. Rate-limit safe: each suite polls with its own bot user (~20 req/min against the 60/min per-user orbit_query limit).
  • MR image path: CARGO_PROFILE_DEV_DEBUG=0 drops DWARF from the dev build; backtraces keep function names via the symbol table. sccache takes a one-time cache-key miss.
  • interruptible: true: superseded pipelines cancel and free cluster capacity; after_script (teardown) runs on cancel on current runners. The pre-existing leak classes (job timeout, runner death) are unchanged and belong to the namespace-reaper follow-up in #967 (closed).

Follow-ups tracked in #967 (closed): warm/persistent GitLab stack across runs (the path to a guaranteed sub-10 even on cold days), stale-namespace TTL reaper in the harness repo, cached CDC-config generation keyed on the gitlab/siphon pins.

Robot Framework bumped 7.2.2→7.4.2 (current latest; requests 0.9.7 and pabot 5.2.2 already latest); pip resolves the set cleanly and the e2e run on this MR exercises it.

%{all_commits}

Edited by Michael Angelo Rivera

Merge request reports

Loading
Loading