<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/engineering/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=628679) </details> <!--IssueSummary end--> ## Summary Roll out [local billable usage persistence](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/18551) currently behind the `local_billing_persistence` feature flag. - DRI: @vij - Team Slack channel: `#g_fulfillment_cost_management` > [!important] > **This flag cannot be rolled out on GitLab.com.** It gates on `License.current&.offline_cloud_license?`, so it is inert on .com, on Dedicated, and on any connected instance without an offline licence. The standard chatops percentage rollout does not apply, and neither do the `gprd` Kibana links or `dashboards.gitlab.net` — an air-gapped instance sends us nothing by definition. The sections below are adapted accordingly. > [!note] > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## Depends on `billing_event_tracking` This flag chooses *where* a billable event goes; it does not decide whether one is captured at all. `Gitlab::BillingEvents::Client#record_usage` checks `billing_event_tracking` on its first line and returns before reaching either branch, so the two are nested rather than independent — the local-persistence path only comes into play once billing event capture is already on. `billing_event_tracking` covers the whole billing events API, arrived in %19.2 via !240688, and is owned by ~"group::analytics instrumentation". It is still default-off, so both need to be on for anything to be recorded: ```ruby Feature.enable(:billing_event_tracking) Feature.enable(:local_billing_persistence) ``` An administrator who enables only this flag sees no rows and no error, so it is worth spelling out in any enablement instructions. - [ ] Check in with ~"group::analytics instrumentation" on `billing_event_tracking` sequencing, so the two rollouts line up ## What could go wrong? **Emission is replaced, not supplemented.** On an offline-licensed instance the client stops emitting to the billing collector and writes to `billable_usage_daily_aggregates` instead. That is the intent for a genuinely air-gapped instance, which was losing the events anyway. It is *not* obviously right for a **connected instance running an offline licence** — that instance emits successfully today and would switch to local rows requiring a manual export. Contained while the flag is opt-in per instance; it must be settled before `default_enabled` is flipped. **Failures are silent.** `track_billing_event` swallows exceptions into error tracking, so a misconfigured instance looks identical to a working one from the caller's side. **Two known limitations**, both recorded on !249324: a gauge cannot report zero, and accumulated `quantity` is not checked against the Iglu export ceiling. ## Verification There is no central telemetry. Verify on the instance: ```ruby # Is the path actually live? Gitlab::BillingEvents::Client.local_persistence_enabled? # => true # Are rows landing? Utilization::BillableUsage::DailyAggregate.order(:id).pluck( :usage_date, :event_type, :feature_qualified_name, :root_namespace_id, :operation_type, :quantity, :events_count) ``` In the instance's own logs (not `gprd`): - `BillingEvents: aggregate not recorded` — the write was rejected; the usual cause is a missing `metadata[:feature_qualified_name]` - `BillingEvents: billing event tracked` — **should be absent**; its presence means emission happened instead of local persistence - `BillingEvents: internal event tracked` — should be present on both paths ## Rollout - [ ] Enable on an offline-licensed test instance and confirm rows land for `secrets_read` (counter) and `secrets_stored` (snapshot) - [ ] Confirm a repeated snapshot supersedes rather than sums, since that is the behaviour unique to this change - [ ] Enable for one willing air-gapped customer with Support, and verify the exported figures against their own expectations - [ ] Decide with Product whether `default_enabled: true` ships in a release, or whether this stays administrator-enabled until the export flow (&23205) lands ## Before defaulting on - [ ] Resolve the connected-instance-with-offline-licence case above - [ ] Export flow (&23205) and ingestion (&23206) are ready, otherwise instances accumulate rows nobody can submit - [ ] Retention decided — rows are currently kept indefinitely (#18555) - [ ] Administrator-facing documentation exists; the current page is developer documentation only ## Rollback Per instance, in a Rails console: ```ruby Feature.disable(:local_billing_persistence) ``` Emission resumes immediately. Rows already written stay in place and remain exportable; nothing needs unwinding. ## Cleanup Remove the flag once the export flow is shipped and the behaviour is stable — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Remove the flag and its YAML definition, then open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup).