Cap the analytics write buffers
Part of Project Bulkhead. Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636120. Builds on !266165 (merged) (merged), which adds the pending gauge these counters sit next to.
What does this MR do and why?
The two Redis lists behind ai_usage_events, PostgreSQL-bound and ClickHouse-bound, get a cap, behind the analytics_write_buffer_protection feature flag (default off). The reasoning is in the confidential issue linked below.
- PostgreSQL buffers.
Analytics::DatabaseWriteBuffer#adddrops rows pastMAX_PENDING(500,000) and counts them ingitlab_database_write_buffer_dropped_total{buffer}. The cap sits in the base class, so every list-shaped buffer gets it; today that isai_usage_events, whose legacy subclass now only keeps its key prefix, andai_token_daily_usages. The hash-shaped buffers (ai_user_metricsand the two dependency-firewall buffers) overrideadd, are bounded by distinct keys rather than event volume, and are not capped. A dropped row does not change the result ofAi::UsageEvent#store_to_pg, soAiTrackingstill records the event's token usage inai_token_daily_usages. - ClickHouse buffer.
ClickHouse::WriteBuffer.adddrops events once a table's list holdsMAX_PENDING(500,000) and counts them ingitlab_clickhouse_write_buffer_dropped_total{table}. For this rollout the cap covers onlyai_usage_events(CAPPED_TABLES);ai_audit_eventsand every other table stay uncapped, because a dropped audit event has no retry or PostgreSQL fallback. The flag definition lives underconfig/because this buffer is CE code. - Requeues bypass the cap. The cap applies to new admissions only. When a drain batch fails, the worker puts it back through
Analytics::DatabaseWriteBuffer#requeue(and the ClickHouse dump worker throughClickHouse::WriteBuffer.add_all, used by !267085 (merged)), which push without the length check, so rows that were already accepted are never dropped because the list filled up while they were out. Theai_user_metricshash buffer's HSETNX put-back moves to its ownrequeuefor the same reason, and so does the failed-batch put-back inAi::TokenDailyUsages::FlushBufferService.
The cap is set well above the backlog a healthy drain leaves; it is a stop against runaway growth, not a throttle, and the pending gauges show how close normal operation gets to it before the flag is enabled. Each check reads the list length and then pushes, so concurrent adds can overshoot by their combined size, which is at most 50 rows per track_events request and one row per event raised inside Rails: a few hundred rows against a cap of 500,000.
An earlier revision also retried a failed batch one row at a time with a dead-letter list. Review asked to keep this MR to the caps: events are validated before they are buffered (Ai::UsageEvent#store_to_pg) and again at dump time, and neither ai_usage_events nor ai_user_metrics carries a foreign key or check constraint, so there is no row-level refusal left for a fallback to catch. A failed batch still goes back to the tail and raises, now through the requeue path so the cap cannot discard it; the far-future timestamp cause is handled in https://gitlab.com/gitlab-org/gitlab/-/work_items/635328.
Business impact
The failure mode this guards against, and its impact, are described in the confidential issue https://gitlab.com/gitlab-org/gitlab/-/work_items/636120.
With analytics_write_buffer_protection enabled (default off), the PostgreSQL list and the ClickHouse list behind ai_usage_events are each capped at 500,000 rows; other ClickHouse tables, including ai_audit_events, are not capped in this rollout. Rows past the cap are dropped and counted in gitlab_database_write_buffer_dropped_total{buffer} and gitlab_clickhouse_write_buffer_dropped_total{table}. Only new events are ever dropped: a failed drain batch is requeued past the cap, so nothing already accepted is lost to it. The user sees nothing (track_events still returns 200); the operator sees the dropped counters rise next to the pending gauges, and AI analytics for that window under-count. Redis availability for every user is what the cap protects.
Queries this change generates
Derived from code. The MR adds no SQL; it issues Redis commands on Gitlab::Redis::SharedState. With the flag on, each add to a capped list does one length check before the push; nothing else changes.
| Buffer | Commands |
|---|---|
ai_usage_events (PostgreSQL) |
LLEN usage_event_write_buffer_ai/usage_event, then RPUSH usage_event_write_buffer_ai/usage_event <json> ... with only the rows under the cap (skipped when all are dropped) |
ai_usage_events (ClickHouse) |
LLEN clickhouse_write_buffer_ai_usage_events, then RPUSH clickhouse_write_buffer_ai_usage_events <json> (skipped at the cap) |
| Other ClickHouse tables | RPUSH clickhouse_write_buffer_<table> <json>, as today |
The drains (LPOP x1,000, INSERT ... FORMAT CSV) are unchanged; a failed batch goes back with one RPUSH of the whole batch, without a length check. Flag off: RPUSH only, as today.
Survivable threshold
Written down here per the Phase 1 definition of done: each capped list holds 500,000 rows. The PostgreSQL dump moves at most 1,000 rows per upsert for up to 200 s every 5 minutes, so the cap is roughly an hour of fully stalled ingestion at the busiest observed rate before rows are dropped; the ClickHouse dump runs every 2 minutes with the same batch size, so its headroom is larger. The pending gauges are the input for tuning both after the flag has run on staging.
Rollout
Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636977. Staging first, watching the two dropped counters against the pending gauges; rollback is turning the flag off, which restores the previous behaviour on the next add with nothing to migrate.
Verification
ee/spec/lib/analytics/database_write_buffer_spec.rb,ee/spec/lib/analytics/legacy_ai_usage_database_write_buffer_spec.rb,ee/spec/lib/analytics/ai_user_metrics_database_write_buffer_spec.rb,spec/lib/click_house/write_buffer_spec.rb,ee/spec/workers/usage_events/dump_write_buffer_cron_worker_spec.rb,ee/spec/workers/analytics/dump_ai_user_metrics_write_buffer_cron_worker_spec.rb: 53 examples, pass locally.- New examples cover, for each buffer: rows past the cap dropped and counted, one row over the cap, a requeued batch kept past the cap, the flag turned on over a backlog already past the cap, flag off adds everything; the ClickHouse cap applies per table and leaves
ai_audit_eventsuncapped past the limit; the legacy buffer keeps its key prefix and inherits the cap; both workers put a failed batch back throughrequeue, and the usage events worker keeps it when the buffer filled up while the batch was out.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.