Keep failed AI user metrics batches and report write buffer backlog

Part of Project Bulkhead. Issues: https://gitlab.com/gitlab-org/gitlab/-/work_items/636341 (fix), https://gitlab.com/gitlab-org/gitlab/-/work_items/636120 (observability part)

What does this MR do and why?

Two small, behaviour-preserving changes to the PostgreSQL write buffers that feed ai_usage_events and ai_user_metrics.

  1. A failed AI user metrics batch is kept instead of lost. Analytics::WriteBufferProcessorWorker puts a failed batch back with model.write_buffer.add(batch), passing an array. Analytics::AiUserMetricsDatabaseWriteBuffer#add only accepted one hash and read attributes[:user_id], so the re-add raised TypeError after pop had already removed the fields from Redis. Up to 1,000 users' last_duo_activity_on updates went missing per failed batch. add now takes one hash or an array, keyed by user_id either way (string or symbol keys, since popped rows come back with string keys). The array path uses HSETNX, so a value tracked for the same user between the pop and the failure is not overwritten by the stale popped row.
  2. The two buffers report their backlog. Each dump run reads the pending count before draining, exports it as the gitlab_database_write_buffer_pending gauge labelled by table (ai_usage_events, ai_user_metrics), and logs it as pending_rows on the done line. This copies the existing gitlab_clickhouse_write_buffer_pending gauge. Until now there was no metric for either PostgreSQL buffer.

No behaviour changes on the happy path. The follow-up MR adds the controls themselves (a cap on the list and per-row fallback with a dead letter for rows that fail on their own), behind a feature flag.

Verification

  • ee/spec/lib/analytics/database_write_buffer_spec.rb, ee/spec/lib/analytics/ai_user_metrics_database_write_buffer_spec.rb, ee/spec/workers/analytics/dump_ai_user_metrics_write_buffer_cron_worker_spec.rb, ee/spec/workers/usage_events/dump_write_buffer_cron_worker_spec.rb: 29 examples, pass locally.
  • New examples: an array of popped rows round-trips through the user metrics buffer; a failing upsert in DumpAiUserMetricsWriteBufferCronWorker leaves all three rows in Redis and raises the original error; the gauge is set with the table label and the pending count.
  • The alert on the new gauge is a runbooks change and will follow once the gauge has production data to set a threshold from.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Chandra Saripaka

Merge request reports

Loading
Loading