Keep failed ClickHouse write buffer batches and stop overlapping dump runs

Part of Project Bulkhead Phase 2. Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636342

What does this MR do and why?

ClickHouse::DumpWriteBufferWorker pops up to 1,000 rows from the Redis write buffer and inserts them into ClickHouse. The pop is destructive, so when the insert raises the popped rows are gone, and nothing stops two runs for the same table from executing at the same time because the cron worker enqueues a job per table every two minutes while a run can last up to MAX_RUNTIME.

Behind the clickhouse_write_buffer_dump_protection feature flag this MR:

  • Pushes the rows of a failed batch back to the buffer before re-raising, so Sidekiq retries with the rows still queued. When a batch holds several field sets, only the groups that have not been inserted yet are re-added, so an earlier successful insert in the same batch is not duplicated. The re-added count is logged as requeued_rows and counted in the new gitlab_clickhouse_write_buffer_requeued_total{table} metric.
  • Takes a per-table exclusive lease (ClickHouse::DumpWriteBufferWorker:<table>, TTL MAX_RUNTIME + 1.minute) around the dump and returns early with status: skipped when another run holds it. The cron interval and the deduplication strategy are unchanged.

With the flag disabled the worker behaves exactly as before: a failed insert still raises and drops the popped rows, and runs are not serialised. ClickHouse::WriteBuffer.add_all is a new bulk RPUSH used only by the re-add path; it does not touch the existing add and pop methods, and rows it puts back are never dropped.

Which failures put the batch back

Only errors that prove ClickHouse did not commit the batch re-add it (REQUEUE_ERRORS): a rejecting response (ClickHouse::Client::DatabaseError, raised on any non-2xx status) or a connection that was never opened (ECONNREFUSED, EHOSTUNREACH, ENETUNREACH, EADDRNOTAVAIL, Net::OpenTimeout, SocketError). A read timeout or a reset after the request body was sent is ambiguous: with wait_for_async_insert=1 the server may already have committed the rows, and because ai_usage_events_daily_mv and code_suggestion_events_daily_mv write 1 AS occurrences into SummingMergeTree tables, re-adding a committed batch would permanently over-count. Those errors therefore keep today's behaviour (rows dropped, error raised). The trade-off is that a slow ClickHouse still loses batches; a fast or unreachable one no longer does.

If putting the rows back fails as well (for example Redis is unreachable), the worker tracks that exception with the table and row count in Gitlab::ErrorTracking, logs lost_rows, and still raises the original insert error so the job retries as before. A failure in the bookkeeping after a successful re-add (counter or log) is tracked and swallowed, so it can neither be reported as lost rows nor replace the insert error.

Lease notes

The lease is released in ensure, so it only leaks when the process dies hard; the next run for that table is then skipped for at most LEASE_TTL (260 s) and logs status: skipped. That delay is accepted for this MR and no separate skipped counter is added.

Overlap with other work

!266191 (merged) (buffer size cap) also edits lib/click_house/write_buffer.rb and its spec. The new method and examples here sit away from its hunks and bypass its cap on purpose; whichever merges second needs a trivial rebase.

Business impact

Today a failed ClickHouse insert loses the up-to-1,000 rows the run had already popped from Redis; one 5xx slot during a deploy leaves a permanent gap in every analytics surface built on ai_usage_events, with nothing but a failed Sidekiq job to show for it. Two runs for the same table can also overlap, because the cron fires every two minutes and a run lasts up to 200 s.

With clickhouse_write_buffer_dump_protection enabled (default off), rows from a batch ClickHouse provably did not commit go back to the buffer before the error is re-raised, so the retry still has them, and a per-table lease keeps runs from overlapping. Operators see requeued_rows in the job log and gitlab_clickhouse_write_buffer_requeued_total{table}; a skipped run logs status: skipped; a failed push-back logs lost_rows and tracks the exception. Read timeouts after the body was sent still drop the batch, because re-adding a committed batch would double count the SummingMergeTree daily views.

Queries this change generates

Derived from code; Redis commands on Gitlab::Redis::SharedState, ClickHouse over HTTP. With the flag on:

SET gitlab:exclusive_lease:ClickHouse::DumpWriteBufferWorker:<table> <uuid> NX EX 260   -- lease for the run
LPOP clickhouse_write_buffer_<table>  x1,000 (pipelined)                                 -- unchanged
INSERT INTO <table> (<fields>) SETTINGS async_insert=1, wait_for_async_insert=1 FORMAT CSV  -- unchanged
LLEN clickhouse_write_buffer_<table>                                                      -- unchanged, pending gauge after the loop
RPUSH clickhouse_write_buffer_<table> <json> ...   -- only after a REQUEUE_ERRORS failure: the groups not yet inserted (ClickHouse::WriteBuffer.add_all)
EVAL <cancel script> ... <uuid>                     -- lease release in ensure

Flag off: LPOP, INSERT, LLEN only, as today. The lease has no renew; it is held for one run and cancelled in ensure.

Threshold

  • No batch lost on an insert failure that ClickHouse rejected or never received.
  • At most one dump run per table at a time.

Rollout

  • Rollout issue: #637721
  • The flag is gitlab_com_derisk and off by default. Rollback is turning the flag off; no data migration is involved.

Verification

bundle exec rspec ee/spec/workers/click_house/dump_write_buffer_worker_spec.rb spec/lib/click_house/write_buffer_spec.rb: 30 examples, 0 failures.

  • ee/spec/workers/click_house/dump_write_buffer_worker_spec.rb: 20 examples, covering flag on with a rejected insert (rows back in original order, counter incremented, error re-raised), a refused connection (rows back), a read timeout (rows not re-added, error re-raised), a failing re-add (lost rows tracked, insert error re-raised), a failing counter after a successful re-add (tracked, rows still queued, insert error re-raised), a failing second field group (only that group re-added), a lease held by another run (skipped, no pop, no insert), a normal run, and flag off (insert failure still raises and rows are not re-added).
  • spec/lib/click_house/write_buffer_spec.rb: 10 examples, including the new .add_all.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Chandra Saripaka

Merge request reports

Loading
Loading