Add Siphon internal events table

What does this MR do and why?

This MR creates the siphon_internal_events table to hold replication Siphon statistic in ClickHouse. With this table we can periodically calculate the Siphon data lag per table:

 SELECT
      postgresql_schema,
      postgresql_table,
      argMax(dateDiff('millisecond', produced_at, consumed_at), consumed_at) AS data_lag_ms
  FROM siphon_internal_events
  WHERE event_type = 1
    AND postgresql_schema <> ''
    AND postgresql_table <> ''
  GROUP BY postgresql_schema, postgresql_table
   ┌─postgresql_schema─┬─postgresql_table──────┬─data_lag_ms─┐
1. │ public            │ issues                │         493 │
2. │ public            │ merge_request_metrics │         256 │
3. │ public            │ projects              │         979 │
4. │ public            │ merge_requests        │         520 │
   └───────────────────┴───────────────────────┴─────────────┘

Why is this important?

Querying data from a different data stores has different consistency and timing guarantees. By knowing the data lag by table, we could do the following:

  • Application may decide to use a different data source (PostgreSQL) in case the CH data lag is high.
  • Provide lag data popover on our visualizations, explaining that the data is not 100% up to date. (For example "red" exclamation mark when the lag goes above 1h).

Schema

column description
uuid Unique identifier for the event, generated by the consumer when the metadata is recorded.
stream_identifier The consumer stream (streams[].identifier) the package was consumed from.
producer_application_identifier application_identifier of the producer that published the package.
consumer_application_identifier application_identifier of the consumer that processed the package.
postgresql_schema Source PostgreSQL schema of the replicated table.
postgresql_table Source PostgreSQL table.
timestamp When the consumer recorded the event (generated locally, UTC). Also the partition/TTL key.
produced_at When the producer published the package to NATS (from the siphon_stream_created_at header). Falls back to consumed_at when the header is absent (e.g. the snapshot-finished marker).
consumed_at When the consumer finished processing the package (same instant used for the siphon_data_lag_ms metric). consumed_at - produced_at is the end-to-end lag.
package_events_count Number of row-level events in the package (len(events)).
event_type Package classification; see the event_type values table above.
version ReplacingMergeTree version column; defaults to now64(6) on insert.
deleted ReplacingMergeTree soft-delete flag; defaults to false. Present for engine consistency; the consumer only inserts.

event_type values:

value meaning
1 Standard CDC package event
2 Initial snapshot data (first event carries the snapshot operation)
3 Snapshot finished
4 ClickHouse refresh event

References

In-product observability about replication stat... (gitlab-org/analytics-section/siphon#177 - closed)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Adam Hegyi

Merge request reports

Loading
Loading