Move OpenBao audit log processing to a Sidekiq worker

Dmytro Biryukovrequested to merge
feature/move-gsm-audit-logs-to-worker-mr-606125 into master

What

Move the OpenBao HTTP audit-device ingest pipeline off the request path of POST /api/v4/internal/secrets_manager/audit_logs:

  • New SecretsManagement::AuditLogWorker performs the exact block the endpoint ran inline until now: SecretsManagement::AuditLog#log! followed by SecretsManagement::BillableEvents::SecretsReadEmitter.emit! (with the emitter rescue preserved so a billing failure never fails/retries the job and double-writes the audit event).
    • urgency :low, data_consistency :delayed, idempotent!, deduplicate :until_executed (identical raw payloads only come from OpenBao re-sends), and defer_on_database_health_signal on audit_events, project_audit_events, group_audit_events.
  • The endpoint validates auth, skips type: "request" lines (OpenBao emits a request-type and a response-type line per operation; request-type lines are always discarded by the pipeline), enqueues the raw JSON, and returns 202 Accepted — which the endpoint already returned while doing everything synchronously.
  • Gated behind the new default-off gitlab_com_derisk feature flag secrets_manager_async_audit_logs; the synchronous path remains as fallback.

Why

OpenBao blocks each of its API responses on this audit round-trip (5s timeout). The synchronous pipeline runs ~10 sequential queries per auditable line — two INSERTs on the main primary (audit_events legacy + project_audit_events/group_audit_events), three root_ancestor namespace traversals, license/plan checks, a routes lookup, and streaming-destination checks. Investigation (see the issue) showed:

  • no single slow query exists — the cost is cumulative, and the query count multiplies exposure to transient primary-DB stalls;
  • slow pipeline executions cause OpenBao to time out and drop the audit line (lost audit events and lost secrets_read billing events), observed in production.

With this change, acceptance becomes a ~ms Redis enqueue independent of DB health; audit-log loss on timeout is replaced by Sidekiq retries/deferral; and the OpenBao audit timeout can subsequently be lowered from 5s to ~1s to cap the worst-case impact on Secrets Manager response times.

Non-goal: this does not fix the LB-level apdex by itself — the observed LB-vs-Rails latency gap lives outside the Rails handler and is tracked separately in the issue.

Rollout

  1. Deploy (no behavior change, flag off).
  2. Enable secrets_manager_async_audit_logs on gprd, watch secrets_management_audit_log queue latency/error rates and audit-event volume for parity.
  3. Follow-up: lower the OpenBao HTTP audit device timeout; remove the flag and the synchronous path.

Test plan

  • RSpec: ee/spec/workers/secrets_management/audit_log_worker_spec.rb (5 examples) and ee/spec/requests/api/internal/secrets_manager_spec.rb (16 examples, async default + synchronous fallback under flag-off context) — green locally.
  • RuboCop clean on all touched files.

References

Merge request reports