Replicate routes table to ClickHouse via siphon

Michael Angelo Riverarequested to merge
arivera/replicate-routes-table into master

Summary

Part of gitlab-org/orbit/knowledge-graph#388 (closed).

Adds the routes table to the siphon CDC pipeline. The Knowledge Graph uses this to resolve full_path for groups and projects via a direct lookup instead of computing it from traversal_ids.

Files:

  • db/siphon/tables/routes.yml -- siphon YAML config (SSOT for helmfiles auto-generation per !10210)
  • db/click_house/migrate/main/20260401003817_create_siphon_routes.rb -- generated ClickHouse migration

Related: gitlab-org/orbit/knowledge-graph!740 (closed)

How to test locally

bundle exec rake gitlab:clickhouse:migrate:main
clickhouse-client -q 'DESCRIBE siphon_routes'
bundle exec rspec spec/db/clickhouse_siphon_tables_spec.rb
E2E validation with GDK VM workspace

Validated using gkg-gdk-vm-workspace.

  • GDK VM with wal_level=logical, siphon, NATS, ClickHouse
  • Created 4-level deep group hierarchy via API: top-org/platform-team/backend/api
  • Siphon replicated all 186 routes to ClickHouse
  • Mid-level rename (platform-team -> engineering): all children cascaded
  • Top-level rename (top-org -> mega-corp): all 4 levels + 3 projects cascaded within 30s
  • Project transfer (toolbox -> gitlab-org): route updated correctly
  • GKG ETL query returns correct full_path at all nesting levels
Edited by Michael Angelo Rivera

Merge request reports