[FF] local_billing_persistence — persist billable usage locally on air-gapped instances

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

Roll out local billable usage persistence currently behind the local_billing_persistence feature flag.

  • DRI: @vij
  • Team Slack channel: #g_fulfillment_cost_management

Important

This flag cannot be rolled out on GitLab.com. It gates on License.current&.offline_cloud_license?, so it is inert on .com, on Dedicated, and on any connected instance without an offline licence. The standard chatops percentage rollout does not apply, and neither do the gprd Kibana links or dashboards.gitlab.net — an air-gapped instance sends us nothing by definition. The sections below are adapted accordingly.

Note

Feature flag controls · Feature flag lifecycle

Depends on billing_event_tracking

This flag chooses where a billable event goes; it does not decide whether one is captured at all. Gitlab::BillingEvents::Client#record_usage checks billing_event_tracking on its first line and returns before reaching either branch, so the two are nested rather than independent — the local-persistence path only comes into play once billing event capture is already on.

billing_event_tracking covers the whole billing events API, arrived in %19.2 via !240688 (merged), and is owned by groupanalytics instrumentation. It is still default-off, so both need to be on for anything to be recorded:

Feature.enable(:billing_event_tracking)
Feature.enable(:local_billing_persistence)

An administrator who enables only this flag sees no rows and no error, so it is worth spelling out in any enablement instructions.

What could go wrong?

Emission is replaced, not supplemented. On an offline-licensed instance the client stops emitting to the billing collector and writes to billable_usage_daily_aggregates instead. That is the intent for a genuinely air-gapped instance, which was losing the events anyway. It is not obviously right for a connected instance running an offline licence — that instance emits successfully today and would switch to local rows requiring a manual export. Contained while the flag is opt-in per instance; it must be settled before default_enabled is flipped.

Failures are silent. track_billing_event swallows exceptions into error tracking, so a misconfigured instance looks identical to a working one from the caller's side.

Two known limitations, both recorded on !249324 (merged): a gauge cannot report zero, and accumulated quantity is not checked against the Iglu export ceiling.

Verification

There is no central telemetry. Verify on the instance:

# Is the path actually live?
Gitlab::BillingEvents::Client.local_persistence_enabled?   # => true

# Are rows landing?
Utilization::BillableUsage::DailyAggregate.order(:id).pluck(
  :usage_date, :event_type, :feature_qualified_name, :root_namespace_id,
  :operation_type, :quantity, :events_count)

In the instance's own logs (not gprd):

  • BillingEvents: aggregate not recorded — the write was rejected; the usual cause is a missing metadata[:feature_qualified_name]
  • BillingEvents: billing event tracked — should be absent; its presence means emission happened instead of local persistence
  • BillingEvents: internal event tracked — should be present on both paths

Rollout

  • Enable on an offline-licensed test instance and confirm rows land for secrets_read (counter) and secrets_stored (snapshot)
  • Confirm a repeated snapshot supersedes rather than sums, since that is the behaviour unique to this change
  • Enable for one willing air-gapped customer with Support, and verify the exported figures against their own expectations
  • Decide with Product whether default_enabled: true ships in a release, or whether this stays administrator-enabled until the export flow (&23205) lands

Before defaulting on

  • Resolve the connected-instance-with-offline-licence case above
  • Export flow (&23205) and ingestion (&23206) are ready, otherwise instances accumulate rows nobody can submit
  • Retention decided — rows are currently kept indefinitely (#18555)
  • Administrator-facing documentation exists; the current page is developer documentation only

Rollback

Per instance, in a Rails console:

Feature.disable(:local_billing_persistence)

Emission resumes immediately. Rows already written stay in place and remain exportable; nothing needs unwinding.

Cleanup

Remove the flag once the export flow is shipped and the behaviour is stable — see cleaning up. Remove the flag and its YAML definition, then open a follow-up Feature Flag Cleanup issue.

Edited by 🤖 GitLab Bot 🤖