Emit secrets_read billable event from the Rails OpenBao audit endpoint

Dmytro Biryukovrequested to merge
feature/emit-secrets-read-billable-event-mr-600967 into master

What

Emit a secrets_read billable usage event from the Rails OpenBao HTTP audit-backend ingest endpoint, using the merged billing module (Gitlab::BillingEvents::Client, !240688 (merged)). Emission is SaaS-only and gated behind a default-off gitlab_com_derisk feature flag (secrets_manager_emit_secret_read_events).

Why

This MR is the producer side of an already-aligned contract. The CDot consumer (https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/15627, merged into 19.0) already registers secrets_read as a billable event type, routes it through Billing::Conversion::UsageCreditCalculator with the quantity_based_rate category, and ships a placeholder credit ratio in config/billing/pricing_multipliers.yml. CDot prices these events the moment Rails starts emitting them.

Originally this MR introduced a bespoke Gitlab::Tracking::BillableUsageTracker

  • Gitlab::Tracking::BillableRealm on top of Gitlab::InternalEvents. Since then the dedicated billing module landed on master (!240688 (merged)), so this MR is rewritten to call it directly and the bespoke tracker/realm classes are dropped. The billing client owns realm/deployment_type, subject/global_user_id, and deterministic event_id generation, and is independently gated by Feature.enabled?(:billing_event_tracking, :instance).

How

  • Emitter — SecretsManagement::BillableEvents::SecretsReadEmitter classifies the parsed SecretsManagement::AuditLog and emits via Gitlab::BillingEvents::Client.track_billing_event. Guard chain in emit!:

    • saas? — ::CloudConnector.gitlab_realm == ::CloudConnector::GITLAB_REALM_SAAS (SM/Dedicated revert to flat-rate pricing, so the monolith emits nothing there).
    • feature_enabled? — the gitlab_com_derisk FF, scoped to the project/group/:instance.
    • billable? — drops type: "request" pre-operation log entries, restricts to secrets_manager_read_project_secret / secrets_manager_read_group_secret, and requires auth.policy_results.allowed == true (successful reads only).
    • return unless billing_namespace — the client reads namespace.id unconditionally; billing_namespace is project.namespace (project mount) or the group (group mount).

    Call arguments: event_type: "secrets_read", category: self.class.name, unit_of_measure: "request", quantity: 1, namespace, project, user: audit_log.author, idempotency_key: "secrets_read:<request.id>", timestamp (parsed from the audit log time), and a compacted metadata hash (pipeline_id, job_id, mount_type, audit_request_id, openbao_entity_id). A rescue StandardError reports to error tracking and swallows, so a billing-emit failure never propagates.

  • Synchronous emission, inline at the API request — hooked from the Grape endpoint POST /api/v4/internal/secrets_manager/audit_logs after audit_log.log!. No worker, no queue — the Snowplow call is an in-process buffer append. A defense-in-depth begin/rescue around the emit call ensures a billing-emit failure never turns into a non-2xx response (which would trigger an OpenBao HTTP-audit-backend retry and double-bill).

  • Deduplication — the OpenBao request.id UUID rides on the event as the idempotency_key. The billing client derives event_id = uuid_v5(instance_uuid, idempotency_key), so a replay of the same OpenBao request collapses to one billing event downstream. The client's own Rails → billing-endpoint retry handling is orthogonal to OpenBao → Rails retries; only the caller can see the upstream key, which is why we supply it.

The captured-from-GDK fixtures (audit_log_kv_data_read_success.json, audit_log_kv_data_read_group_secret_success.json, audit_log_kv_read_permission_denied.json) pin these field names against actual OpenBao output and close a real fixture gap in the repo.

Out of scope

  • Analytics / Service Ping — the billing module emits its own usage_billing_event internal event in a follow-up Analytics MR; the Service Ping metric is re-added there on top of that event. No event YAML or metric definition ships here.
  • Non-SaaS emission — pending a trusted, GitLab-owned emission path for self-managed.

Coordination with sibling MR !238145 (merged) (secrets_stored)

The earlier plan had this MR introduce the shared BillableUsageTracker / BillableRealm for !238145 (merged) to consume. That coupling is gone: both events now target the merged billing module directly, so there is no longer a hard merge-order dependency between the two MRs.

Test plan

  • RuboCop clean on touched files.
  • RSpec green: emitter spec (project / group / permission-denied / FF-off / SM-Dedicated / request-log / non-read / raise-swallowed) and the amended request spec (API → emitter wiring + end-to-end SaaS billing call).
  • Real OpenBao payloads captured from a local GDK with the file audit device enabled; the JSON fixtures are derived from that capture (tokens redacted, PII placeholdered).
  • End-to-end against a GDK with the HTTP audit device pointed at the Rails ingest endpoint, both FFs on, verifying the billable_usage event lands in Snowplow Micro.
  • Database review N/A — no migrations, no schema changes.

Screenshots

N/A — no UI change.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

References

Edited by Dmytro Biryukov

Merge request reports