feat(xtask): clickhouse setup and migrations for e2e harness

What does this MR do and why?

Adds the first part of Phase 3 ("GKG stack") to the cargo xtask E2E runner: deploying ClickHouse and applying both the datalake and graph schemas (steps 15-17). ClickHouse must be running before Tilt starts Siphon because the materialized views only fire on new inserts — if the tables don't exist when data flows in, it's silently lost.

Also hardens all bash -c shell-outs across the xtask crate against injection (positional parameters instead of string interpolation), and extracts 16 inline constants (Helm release names, repo URLs, timeouts, label selectors) into constants.rs to eliminate cross-file duplication.

Continues from !314 (merged) (Phase 2, pending merge). Ports from !304 (closed).

What changed

  • gkg.rs — steps 15-17: deploy ClickHouse via static manifests, run datalake migrations (gitlab:clickhouse:migrate via kubectl cp + rake), apply graph schema (graph.sql via stdin pipe to clickhouse-client)
  • e2e/cng/clickhouse.yaml — ConfigMap + Service + StatefulSet for standalone ClickHouse
  • constants.rs — ClickHouse config defaults + extracted Helm/Traefik/PG constants from inline usage across cng.rs, cngsetup.rs, teardown.rs
  • kubectl.rsget_ch_pod helper; hardened all bash -c patterns in pg_superuser_exec, toolbox_rails_eval
  • config.rs — ClickHouse fields (service name, URL, datalake/graph DB names)
  • main.rs--gkg flag now wired to gkg::run

How to validate locally

# Run all phases including GKG stack (steps 15-17)
GITLAB_SRC=~/path/to/gitlab cargo xtask e2e setup --gkg

# GKG stack only (cluster + GitLab already running)
# (not yet a standalone flag — use --gkg which runs all phases)

References

  • Continues: !314 (merged) (Phase 2 — CNG setup, pending merge)
  • Derived from: !304 (closed) (shell-based E2E harness)
  • Next: MR 3b (steps 18-24: siphon reset, Tilt, dispatch-indexing, optimize, verify graph tables)

Testing

Performance Analysis

  • This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Edited by Michael Usachenko

Merge request reports

Loading