Add resource_state_events to Siphon replication

What does this MR do and why?

Adds the Rails resource_state_events table to Siphon CDC replication into the ClickHouse datalake, creating siphon_resource_state_events.

This unblocks the GitLab Knowledge Graph work for REOPENED edges (gitlab-org/orbit/knowledge-graph#883 (closed)), which cannot be emitted until resource_state_events is available in ClickHouse.

The change mirrors the precedent merge request !237936 (merged) ("Add system_note_metadata to Siphon replication"). The table has a direct namespace_id column (NOT NULL), so the traversal_path derivation is a direct copy of the precedent using dictGetOrDefault('namespace_traversal_paths_dict', ...) with no indirect foreign-key resolution.

All ClickHouse artifacts (migrations, schema cache, schema-migration version stamps, and db/click_house/main.sql) were produced by the repository tooling, not hand-written:

rails generate gitlab:click_house:siphon resource_state_events --with-traversal-path
rake gitlab:clickhouse:migrate:main

Column mapping (Rails → ClickHouse)

Rails column Type ClickHouse
id bigint NOT NULL Int64 (PK)
user_id bigint Nullable(Int64)
issue_id bigint Nullable(Int64)
merge_request_id bigint Nullable(Int64)
created_at timestamptz NOT NULL DateTime64(6, 'UTC')
state smallint NOT NULL Int16
epic_id bigint Nullable(Int64)
source_commit text Nullable(String)
close_after_error_tracking_resolve boolean NOT NULL Bool
close_auto_resolve_prometheus_alert boolean NOT NULL Bool
source_merge_request_id bigint Nullable(Int64)
imported_from smallint NOT NULL Int16
namespace_id bigint NOT NULL Int64

Testing / validation (local, full GDK with ClickHouse)

  • Migration applies cleanly; siphon_resource_state_events, the _pg_pkey_ordered table, and the materialized view are all created.
  • Reversible: rollback STEP=2 drops all three objects, then migrate recreates them.
  • spec/db/clickhouse_siphon_tables_spec.rb → 56 examples, 0 failures (PG ↔️ ClickHouse column parity for every siphon table).
  • spec/scripts/click_house/schema_validator_spec.rb → 46 examples, 0 failures.
  • ClickHouse::SchemaValidator.validate! → "Schema is up to date - no changes detected".
  • RuboCop on both migrations → no offenses.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Dmitry Gruzd

Merge request reports

Loading
Loading