feat(xtask): clickhouse setup and migrations for e2e harness
What does this MR do and why?
Adds the first part of Phase 3 ("GKG stack") to the cargo xtask E2E runner: deploying ClickHouse and applying both the datalake and graph schemas (steps 15-17). ClickHouse must be running before Tilt starts Siphon because the materialized views only fire on new inserts — if the tables don't exist when data flows in, it's silently lost.
Also hardens all bash -c shell-outs across the xtask crate against injection (positional parameters instead of string interpolation), and extracts 16 inline constants (Helm release names, repo URLs, timeouts, label selectors) into constants.rs to eliminate cross-file duplication.
Continues from !314 (merged) (Phase 2, pending merge). Ports from !304 (closed).
What changed
-
gkg.rs— steps 15-17: deploy ClickHouse via static manifests, run datalake migrations (gitlab:clickhouse:migratevia kubectl cp + rake), apply graph schema (graph.sqlvia stdin pipe toclickhouse-client) -
e2e/cng/clickhouse.yaml— ConfigMap + Service + StatefulSet for standalone ClickHouse -
constants.rs— ClickHouse config defaults + extracted Helm/Traefik/PG constants from inline usage across cng.rs, cngsetup.rs, teardown.rs -
kubectl.rs—get_ch_podhelper; hardened allbash -cpatterns inpg_superuser_exec,toolbox_rails_eval -
config.rs— ClickHouse fields (service name, URL, datalake/graph DB names) -
main.rs—--gkgflag now wired togkg::run
How to validate locally
# Run all phases including GKG stack (steps 15-17)
GITLAB_SRC=~/path/to/gitlab cargo xtask e2e setup --gkg
# GKG stack only (cluster + GitLab already running)
# (not yet a standalone flag — use --gkg which runs all phases)
References
- Continues: !314 (merged) (Phase 2 — CNG setup, pending merge)
- Derived from: !304 (closed) (shell-based E2E harness)
- Next: MR 3b (steps 18-24: siphon reset, Tilt, dispatch-indexing, optimize, verify graph tables)
Related Issues
Testing
Performance Analysis
- This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Edited by Michael Usachenko