Skip to content

Data migration strategy for moving `audit_events` from PG to Clickhouse

Problem Statement

Sync Audit Events from PG to ClickHouse.

Implementation Plan

  1. Write a cron job worker which enqueues all the partitions of audit events into a worker.
  2. Write a worker that takes input from a partition of the audit event table and syncs its data into ClickHouse.
  3. The syncing should happen by making a CSV and should use batching.
  4. The worker should close the CSV and upload at a certain time limit.
  5. And when the job re-runs it should run from where the previous worker left.
Edited by Harsimar Sandhu