Add resource_state_events to Siphon replication
What does this MR do and why?
Adds the Rails resource_state_events table to Siphon CDC replication into the
ClickHouse datalake, creating siphon_resource_state_events.
This unblocks the GitLab Knowledge Graph work for REOPENED edges
(gitlab-org/orbit/knowledge-graph#883 (closed)), which cannot be emitted
until resource_state_events is available in ClickHouse.
The change mirrors the precedent merge request
!237936 (merged)
("Add system_note_metadata to Siphon replication"). The table has a direct
namespace_id column (NOT NULL), so the traversal_path derivation is a direct
copy of the precedent using dictGetOrDefault('namespace_traversal_paths_dict', ...)
with no indirect foreign-key resolution.
All ClickHouse artifacts (migrations, schema cache, schema-migration version stamps,
and db/click_house/main.sql) were produced by the repository tooling, not hand-written:
rails generate gitlab:click_house:siphon resource_state_events --with-traversal-path
rake gitlab:clickhouse:migrate:mainColumn mapping (Rails → ClickHouse)
| Rails column | Type | ClickHouse |
|---|---|---|
id |
bigint NOT NULL |
Int64 (PK) |
user_id |
bigint |
Nullable(Int64) |
issue_id |
bigint |
Nullable(Int64) |
merge_request_id |
bigint |
Nullable(Int64) |
created_at |
timestamptz NOT NULL |
DateTime64(6, 'UTC') |
state |
smallint NOT NULL |
Int16 |
epic_id |
bigint |
Nullable(Int64) |
source_commit |
text |
Nullable(String) |
close_after_error_tracking_resolve |
boolean NOT NULL |
Bool |
close_auto_resolve_prometheus_alert |
boolean NOT NULL |
Bool |
source_merge_request_id |
bigint |
Nullable(Int64) |
imported_from |
smallint NOT NULL |
Int16 |
namespace_id |
bigint NOT NULL |
Int64 |
Testing / validation (local, full GDK with ClickHouse)
- Migration applies cleanly;
siphon_resource_state_events, the_pg_pkey_orderedtable, and the materialized view are all created. - Reversible:
rollback STEP=2drops all three objects, thenmigraterecreates them. spec/db/clickhouse_siphon_tables_spec.rb→ 56 examples, 0 failures (PG↔️ ClickHouse column parity for every siphon table).spec/scripts/click_house/schema_validator_spec.rb→ 46 examples, 0 failures.ClickHouse::SchemaValidator.validate!→ "Schema is up to date - no changes detected".- RuboCop on both migrations → no offenses.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
- I have evaluated the MR acceptance checklist for this MR.