Emit daily secrets_stored billable event from persisted namespace_secret_counts

Dmytro Biryukovrequested to merge
feature/emit-secrets-stored-billable-event-mr-597867 into master

What

Emit a daily secrets_stored billable usage event, one per root namespace, by aggregating the persisted counts in namespace_secret_counts (!234604 (merged)) and dispatching through the merged billing module Gitlab::BillingEvents::Client (!240688 (merged)). SaaS-only, gated behind a default-off gitlab_com_derisk feature flag secrets_manager_emit_secret_stored_events.

Why

This MR is the producer side of the second of the two billable events in the GitLab Secrets Manager pricing contract:

  • secrets_read — per-request, emitted from the Rails OpenBao HTTP audit-backend ingest endpoint (!238134 (merged)).
  • secrets_stored — daily aggregate, emitted from a cron worker that reads the persisted counts (this MR).

The CDot consumer side already registers both event types and routes them through Billing::Conversion::UsageCreditCalculator with the quantity_based_rate category, so the moment Rails starts emitting secrets_stored events, CDot prices them.

Persisting the per-namespace counts in namespace_secret_counts (!234604 (merged)) means this MR is a pure read path against PostgreSQL — no per-emission OpenBao round-trip — so a daily fan-out across all root namespaces is O(distinct root_namespace_ids).

How

  • Billing module, no event YAML — the emitter calls Gitlab::BillingEvents::Client.track_billing_event directly (merged in !240688 (merged)). The client owns the payload: it stamps namespace_id / root_namespace_id from namespace:, stamps timestamp, and derives event_id = uuid_v5(instance_uuid, idempotency_key). No ee/config/events/*.yml definition ships and no InternalEvents call is made.
  • Daily fan-out cron — SecretsManagement::EmitSecretsStoredBillableEventCronWorker registered at 17 3 * * *:
    • Resolves distinct roots via the new NamespaceSecretCount.distinct_root_namespace_ids scope, then ::Namespace.id_in(...).each_batch(of: 500). The each_batch is on namespaces.id (a PK column) so the iteration stays index-driven; the distinct-roots subquery is supported by index_namespace_secret_counts_on_root_namespace_id from !234604 (merged).
    • Bulk-enqueues per batch via bulk_perform_async_with_contexts (the canonical GitLab cron fan-out idiom).
    • Short-circuits on Gitlab::Database.read_only? and on the secrets_manager_emit_secret_stored_events FF at instance scope.
    • Carries defer_on_database_health_signal :gitlab_main_org, [:namespace_secret_counts], 1.minute so a hot/lagging table defers the daily run rather than amplifying load.
  • Per-root emission worker — SecretsManagement::EmitSecretsStoredBillableEventWorker(root_namespace_id):
    • deduplicate :until_executed, including_scheduled: true collapses any re-enqueue of the same root_namespace_id within the daily window into one emission.
    • self.idempotency_arguments strips trailing args so the dedupe key is purely [root_namespace_id].
    • Delegates to SecretsManagement::BillableEvents::SecretsStoredEmitter.emit_for_root_namespace_id! and otherwise does nothing.
  • Emitter — SecretsManagement::BillableEvents::SecretsStoredEmitter (mirrors the shape of SecretsReadEmitter from !238134 (merged); differs in source of truth, cadence, and failure invariant):
    • Guards, in order: SaaS-only (::CloudConnector.gitlab_realm == ::CloudConnector::GITLAB_REALM_SAAS; Self-Managed/Dedicated skip emission entirely) → FF check with the root namespace as actor → skip when quantity == 0.
    • Quantity = SUM(count) over namespace_secret_counts scoped by root_namespace_id. No OpenBao call.
    • Stable idempotency key — secrets_stored:<root_namespace_id>:<YYYY-MM-DD>. The client derives event_id from it, so a same-day re-emission of the same root collapses to one billing event downstream.
    • Metadata distinguishes quantity (SUM of stored secrets across the root) from namespace_count (COUNT of namespaces under the root that carry any rows). Reviewers will ask — they are deliberately different.
    • Double-gated FF, kill-switch only. Cron checks the flag before enqueuing; emitter re-checks before each per-root job runs. Both reduce to "is the flag on?" — neither gives per-tenant rollout. The second check exists so disabling the flag stops any per-root jobs the cron has already enqueued but Sidekiq has not yet picked up.
    • Skips emission entirely when quantity == 0 (and when no rows exist for the root, since SUM returns 0 in that case).
    • Never raises. Bubbling errors would let Sidekiq retry the same root_namespace_id and double-bill the same day. Errors are tracked via Gitlab::ErrorTracking and the method returns nil.
  • Model scope — new NamespaceSecretCount.distinct_root_namespace_ids on the model added in !234604 (merged) keeps the worker free of inline select(...).distinct (and out of CodeReuse/ActiveRecord territory).

Sequencing / dependencies

  • No shared-code dependency anymore. The previous hard dependency on the tracker classes from !238134 (merged) is gone — both producers now call the merged Gitlab::BillingEvents::Client (!240688 (merged), already on master) directly.
  • This MR targets the !238134 (merged) branch only for a clean diff; once !238134 (merged) merges, it will be retargeted to master.
  • Reads from !234604 (merged) (already merged). namespace_secret_counts and the supporting index index_namespace_secret_counts_on_root_namespace_id are exercised by the daily fan-out subquery.

Database notes

No migrations. The only new read path is the daily cron's subquery:

SELECT "namespaces".*
FROM "namespaces"
WHERE "namespaces"."id" IN (
  SELECT DISTINCT "namespace_secret_counts"."root_namespace_id"
  FROM "namespace_secret_counts"
)
ORDER BY "namespaces"."id" ASC
LIMIT 500;

Postgres.ai explain:

  1. Cold cache: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/52473/commands/153976
  2. Hot cache: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/52473/commands/153978

Supported by index_namespace_secret_counts_on_root_namespace_id (added in !234604 (merged)). The per-root SUM(count) WHERE root_namespace_id = ? inside the emitter uses the same index.

Cardinality bound: one row per root namespace with at least one active group/project secrets manager — low thousands cluster-wide today — so the daily fan-out is bounded. data_consistency :sticky is set on both workers to read from the primary when the replica lags, which keeps SUM(count) consistent with whatever ReconcileNamespaceSecretCountsCronWorker last wrote.

Feature flag

Default-off gitlab_com_derisk flag: secrets_manager_emit_secret_stored_events. Rolled out instance-wide for the cron gate, then per-root for the emitter gate, so a partial rollout can target specific tenants without re-enabling the cron fan-out for everyone.

Test plan

  • RuboCop clean on touched files.
  • RSpec covers the emitter (full billing-event kwargs including the date-scoped idempotency key, zero-count, no-rows, FF off, non-SaaS realm, raise-and-swallow), the per-root worker (delegation + idempotency-arguments contract + idempotent shared example), and the cron (distinct-root fan-out, no rows, FF off, read-only DB, idempotent shared example).
  • Local GDK validation (GITLAB_SIMULATE_SAAS=1 console): seeded namespace_secret_counts across a root + subgroup, enabled secrets_manager_emit_secret_stored_events and billing_event_tracking, verified BillingEvents: event tracked with quantity: 8, a stable event_id across same-day re-emissions (idempotency key), cron fan-out enqueuing one job per distinct root, and no-ops for FF off / zero quantity / non-SaaS realm.

Screenshots

N/A — backend-only, no UI.

MR acceptance checklist

References

Edited by Dmytro Biryukov

Merge request reports