Data migration strategy for moving `audit_events` from PG to Clickhouse

Problem Statement

Sync Audit Events from PG to ClickHouse.

Implementation Plan

  1. Write a cron job worker which enqueues all the partitions of audit events into a worker.
  2. Write a worker that takes input from a partition of the audit event table and syncs its data into ClickHouse.
  3. The syncing should happen by making a CSV and should use batching.
  4. The worker should close the CSV and upload at a certain time limit.
  5. And when the job re-runs it should run from where the previous worker left.
Edited Oct 19, 2023 by Harsimar Sandhu
Assignee Loading
Time tracking Loading