Loading
Replicate ai_governance_sessions to ClickHouse with Siphon
What does this MR do and why?
Replicates the new ai_governance_sessions Postgres table to ClickHouse using Siphon, the same approach used for duo_workflows_workflows. The AI Governance dashboard reads from ClickHouse on GitLab.com, with Postgres only as a fallback, so without this replication, rows in the new table would not be visible there.
Notes for reviewers
- Files were generated with
rails generate gitlab:click_house:siphon ai_governance_sessions --with-traversal-path, then hand-edited:- Changed the ClickHouse sort key to
(traversal_path, session_started_at, id)instead of the generator'screated_at, so ClickHouse orders sessions the same way as the Postgres index(namespace_id, session_started_at, id)and the dashboard's Start time column. - Added
session_started_atto the lookup tablesiphon_ai_governance_sessions_pg_pkey_orderedand its materialized view, since the lookup table needs every sort-key column so Siphon can find and replace old rows on update.
- Changed the ClickHouse sort key to
- The reconcile config only lists
namespace_id, because that is the only sharding key defined indb/docs/ai_governance_sessions.yml.
References
- Closes https://gitlab.com/gitlab-org/gitlab/-/work_items/630093
- Epic: https://gitlab.com/groups/gitlab-org/-/epics/21540
- Blocks: https://gitlab.com/gitlab-org/gitlab/-/work_items/630095
- Table added in !256344 (merged) (merged)
Screenshots or screen recordings
Not applicable, this MR has no UI change.
How to set up and validate locally
- Run
bundle exec rspec spec/db/clickhouse_siphon_tables_spec.rb(80 examples, 0 failures locally). - Optionally, with Siphon enabled in GDK, run
gdk reconfigure && gdk restart siphon, then in a rails console runClickHouse::Client.select('SELECT count() FROM siphon_ai_governance_sessions FINAL', :main).
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Hitesh Raghuvanshi