siphon.md

Configure Siphon to run in GDK

You can configure Siphon to run in GDK. Siphon is not yet ready for production use.

Siphon facilitates data synchronization between PostgreSQL and other data stores. You can use Siphon in GDK to synchronize data between tables in PostgreSQL to matching tables in ClickHouse, which allows you to write features that will benefit from the performance improvements of using an OLAP database.

Prerequisites

Before configuring Siphon to run in GDK, you must:

• Enable ClickHouse in your GDK. For more information, see ClickHouse within GitLab.
• Enable NATS in your GDK. gdk config set nats.enabled true
• Configure a Docker runtime. Only Docker Desktop is supported, but other container runtimes might work.

Enable logical replication in PostgreSQL

To use Siphon, you must enable logical replication for GDK's main PostgreSQL database:

1. Open the $GDK_ROOT/postgresql/data/postgresql.conf file.

2. Change wal_level value to wal_level = logical.

3. If $GDK_ROOT/postgresql/data/replication.conf exists, also change wal_level = logical there. (This file is only likely to exist if PostgreSQL had replication enabled at any point)

4. Restart the PostgreSQL service:

   gdk restart postgresql

Start and configure Siphon

To start and configure Siphon:

1. Update gdk.yml to enable Siphon:

   gdk config set siphon.enabled true

2. Run gdk reconfigure to generate configuration files.

3. Optional. In GitLab Rails directory (gitlab), run the Siphon migration generator to generate a table. The migration generator creates a migration to create a table in ClickHouse.

   rails generate gitlab:click_house:siphon users

4. Run the ClickHouse migrations:

   bundle exec rake gitlab:clickhouse:migrate

5. Run gdk start to start the new service.

Validate

To validate that Siphon is working in your GDK:

1. Check that GDK is running the siphon-producer-main-db and siphon-clickhouse-consumer services.

2. Inspect the output of these services with gdk tail siphon-producer-main-db and gdk tail siphon-clickhouse-consumer.

3. Add new data to the provisioned tables (users in this case).

4. Note that the data should be replicated to the siphon_users table in ClickHouse. Validate this using:

   SELECT COUNT(*) FROM siphon_users FINAL;