Skip to content
Snippets Groups Projects

Draft: Make collector batching configurable in cluster CR

Open Mat Appelman requested to merge mappelman/larger-ch-inserts into main

After running some experiments, we've witnessed significant improvements when inserting larger batches into ClickHouse. We've landed on a batching configuration that works well for our current loads.

The goal now of this MR is to make the batching configuration configurable in the Cluster CR in order to configure different values for dev/testing/production.

Edited by Mat Appelman

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
29 29 key: gitlab.target_namespace_id
30 30 from_context: metadata.x-target-namespaceid
31 31 batch:
32 send_batch_size: 25000
33 send_batch_max_size: 50000
34 timeout: 5s
32 send_batch_size: 200000
33 send_batch_max_size: 250000
34 timeout: 10s
  • Mat Appelman added 1 commit

    added 1 commit

    • 30b8ffa5 - Improve CH performance with larger inserts

    Compare with previous version

  • Mat Appelman changed title from Draft: Improve CH performance with larger inserts to Draft: Make collector batching configurable in cluster CR

    changed title from Draft: Improve CH performance with larger inserts to Draft: Make collector batching configurable in cluster CR

  • Mat Appelman changed the description

    changed the description

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading