Loading
Keep failed AI user metrics batches and report write buffer backlog
Part of Project Bulkhead. Issues: https://gitlab.com/gitlab-org/gitlab/-/work_items/636341 (fix), https://gitlab.com/gitlab-org/gitlab/-/work_items/636120 (observability part)
What does this MR do and why?
Two small, behaviour-preserving changes to the PostgreSQL write buffers that feed ai_usage_events and ai_user_metrics.
- A failed AI user metrics batch is kept instead of lost.
Analytics::WriteBufferProcessorWorkerputs a failed batch back withmodel.write_buffer.add(batch), passing an array.Analytics::AiUserMetricsDatabaseWriteBuffer#addonly accepted one hash and readattributes[:user_id], so the re-add raisedTypeErrorafterpophad already removed the fields from Redis. Up to 1,000 users'last_duo_activity_onupdates went missing per failed batch.addnow takes one hash or an array, keyed byuser_ideither way (string or symbol keys, since popped rows come back with string keys). The array path usesHSETNX, so a value tracked for the same user between the pop and the failure is not overwritten by the stale popped row. - The two buffers report their backlog. Each dump run reads the pending count before draining, exports it as the
gitlab_database_write_buffer_pendinggauge labelled by table (ai_usage_events,ai_user_metrics), and logs it aspending_rowson the done line. This copies the existinggitlab_clickhouse_write_buffer_pendinggauge. Until now there was no metric for either PostgreSQL buffer.
No behaviour changes on the happy path. The follow-up MR adds the controls themselves (a cap on the list and per-row fallback with a dead letter for rows that fail on their own), behind a feature flag.
Verification
ee/spec/lib/analytics/database_write_buffer_spec.rb,ee/spec/lib/analytics/ai_user_metrics_database_write_buffer_spec.rb,ee/spec/workers/analytics/dump_ai_user_metrics_write_buffer_cron_worker_spec.rb,ee/spec/workers/usage_events/dump_write_buffer_cron_worker_spec.rb: 29 examples, pass locally.- New examples: an array of popped rows round-trips through the user metrics buffer; a failing upsert in
DumpAiUserMetricsWriteBufferCronWorkerleaves all three rows in Redis and raises the original error; the gauge is set with the table label and the pending count. - The alert on the new gauge is a runbooks change and will follow once the gauge has production data to set a threshold from.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Chandra Saripaka