@bohdanpk, not sure where you get this error. DELETE needs special handling which we already handle in the clickhouse consumer:

1. Find the latest row version by the PG primary keys (you might need a projection to speed this up).
2. Reinsert the row with higher version timestamp + the delete flag.

If it's the ClickHouse consumer then we need to take a closer look.

Ok, I see where it's coming from! 👍

@bohdanpk,

This one and the TOAST issue has been fixed with 0.0.28-beta. One important configuration change that's needed:

If the ClickHouse primary keys and the PostgreSQL primary keys are not exactly the same, we must specify the PostgreSQL primary keys in the consumer config. Generation of this config could be automated if there is connectivity to PG and CH.

What's affected:

• any table that includes the traversal_path column with DEFAULT
• siphon_merge_request_assignees, siphon_approvals, siphon_deployments, siphon_environments, siphon_members (dedup_by needs to be set to id)
• siphon_deployment_merge_requests (deployment_id, merge_request_id)
• (id)
• siphon_project_authorizations (user_id, project_id, access_level)

Using the siphon_knowledge_graph_enabled_namespaces as an example in the consumer:

streams:
  - identifier: knowledge_graph_enabled_namespaces
    subject: knowledge_graph_enabled_namespaces
    target: siphon_knowledge_graph_enabled_namespaces
    dedup_by:
      - id

@ahegyi Thank you for the quick action! We are very carefully following all updates to the siphon and will be keeping the team aware about any issues that we find.