WIP: Resolve "stream changes from airflow DB to snowflake"
Closes #72 (closed)
setting up streaming CDC from the Airflow MetaData DB to our Snowflake instance using debezium
-
update airflow DB to postgres 11 (so that debezium can perform replication) -
turn off stitch cloudsql replication for the Airflow DB
Set up the relevant containers/services:
-
zookeeper -
kafka -
kafka connect source for postgres -
kafka connect sink for snowflake
Configuration for the postgres connector:
-
snapshot whenever a connection is reestablished
Edited by Thomas La Piana