Loading
Add Siphon internal events table
What does this MR do and why?
This MR creates the siphon_internal_events table to hold replication Siphon statistic in ClickHouse. With this table we can periodically calculate the Siphon data lag per table:
SELECT
postgresql_schema,
postgresql_table,
argMax(dateDiff('millisecond', produced_at, consumed_at), consumed_at) AS data_lag_ms
FROM siphon_internal_events
WHERE event_type = 1
AND postgresql_schema <> ''
AND postgresql_table <> ''
GROUP BY postgresql_schema, postgresql_table ┌─postgresql_schema─┬─postgresql_table──────┬─data_lag_ms─┐
1. │ public │ issues │ 493 │
2. │ public │ merge_request_metrics │ 256 │
3. │ public │ projects │ 979 │
4. │ public │ merge_requests │ 520 │
└───────────────────┴───────────────────────┴─────────────┘Why is this important?
Querying data from a different data stores has different consistency and timing guarantees. By knowing the data lag by table, we could do the following:
- Application may decide to use a different data source (PostgreSQL) in case the CH data lag is high.
- Provide lag data popover on our visualizations, explaining that the data is not 100% up to date. (For example "red" exclamation mark when the lag goes above 1h).
Schema
| column | description |
|---|---|
uuid |
Unique identifier for the event, generated by the consumer when the metadata is recorded. |
stream_identifier |
The consumer stream (streams[].identifier) the package was consumed from. |
producer_application_identifier |
application_identifier of the producer that published the package. |
consumer_application_identifier |
application_identifier of the consumer that processed the package. |
postgresql_schema |
Source PostgreSQL schema of the replicated table. |
postgresql_table |
Source PostgreSQL table. |
timestamp |
When the consumer recorded the event (generated locally, UTC). Also the partition/TTL key. |
produced_at |
When the producer published the package to NATS (from the siphon_stream_created_at header). Falls back to consumed_at when the header is absent (e.g. the snapshot-finished marker). |
consumed_at |
When the consumer finished processing the package (same instant used for the siphon_data_lag_ms metric). consumed_at - produced_at is the end-to-end lag. |
package_events_count |
Number of row-level events in the package (len(events)). |
event_type |
Package classification; see the event_type values table above. |
version |
ReplacingMergeTree version column; defaults to now64(6) on insert. |
deleted |
ReplacingMergeTree soft-delete flag; defaults to false. Present for engine consistency; the consumer only inserts. |
event_type values:
| value | meaning |
|---|---|
1 |
Standard CDC package event |
2 |
Initial snapshot data (first event carries the snapshot operation) |
3 |
Snapshot finished |
4 |
ClickHouse refresh event |
References
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Adam Hegyi