Add Siphon replication for workflow notes and work items to ClickHouse

What does this MR do and why?

Adds Siphon replication of the duo_workflows_workflow_notes and duo_workflows_workflow_work_items PostgreSQL tables to ClickHouse.

The AI Audit Events UI needs to surface which notes and work items a Duo Workflow session touched. The merge-request link table (duo_workflows_workflow_merge_requests) is already replicated to ClickHouse via Siphon, but the note and work-item link tables had no Siphon config and no ClickHouse tables.

This mirrors the existing duo_workflows_workflow_merge_requests pattern exactly:

  • Siphon replication configs: db/siphon/tables/duo_workflows_workflow_notes.yml and db/siphon/tables/duo_workflows_workflow_work_items.yml
  • ClickHouse siphon_* tables using ReplacingMergeTree(_siphon_replicated_at, _siphon_deleted) ordered by (traversal_path, workflow_id, id)
  • *_pg_pkey_ordered dedup lookup tables + materialized views for each
  • db/click_house/main.sql schema dump, schema cache YAMLs, and schema-migration checksum files

References

Screenshots or screen recordings

N/A — no user-facing UI changes.

How to set up and validate locally

  1. Run the ClickHouse migrations:
    bundle exec rake gitlab:clickhouse:migrate
  2. Regenerate the schema dump and cache:
    bundle exec rake gitlab:clickhouse:schema:dump
  3. Validate the siphon table configuration via the database spec:
    bundle exec rspec spec/db/clickhouse_siphon_tables_spec.rb

Note: This environment has no local PostgreSQL/ClickHouse, so the migrations and clickhouse:check-schema job must run in CI. The migration files are byte-identical to the already-merged duo_workflows_workflow_merge_requests pattern (verified via diff), and the schema dump / cache / checksums were authored to match formatQuery output exactly.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports

Loading
Loading