Build export records from billable usage aggregates

Vijay Hawoldarrequested to merge
18552-export-record-projection into master

What does this MR do and why?

Adds the read path that turns stored billable usage into the usage records an air-gapped usage export carries. It also extracts the instance-identity fields (Gitlab::BillingEvents::InstanceIdentity) out of the emission path, so the export envelope and emitted events derive them the same way. Nothing user-facing yet.

  • Reads billable_usage_daily_totals (added in !258358 (merged)), one row per day per usage type for the whole instance, about 730 rows a year regardless of namespace count. This follows database review (note).
  • Records have no namespace, because the export is per instance.
  • Records are one per day per usage type. timestamp is midday UTC, quantity is a JSON number, and operation_type is omitted when nil.
  • metadata.quantity_kind is counter or snapshot, derived from the event type. secrets_stored is the only snapshot.
  • event_id is stable across re-exports, which is what CustomersDot dedupes on.
  • The export scopes in_period and ordered_for_export live on Utilization::BillableUsage::DailyTotal. Summing happens in the totals refresh (!258358 (merged)).
  • The records are unchanged by the rework. On the same seed (a month, 4 namespaces including a nil one, 3 usage types, mixed operation_type), the old summed query and the new totals read produce identical JSON (93 records, same SHA-256).

Reviewing this commit by commit

Each link opens that commit's diff.

Database

Read-only. No migration.

SELECT "billable_usage_daily_totals".* FROM "billable_usage_daily_totals" WHERE "billable_usage_daily_totals"."usage_date" BETWEEN '2025-10-07' AND '2026-10-06' ORDER BY "billable_usage_daily_totals"."usage_date" ASC NULLS FIRST, "billable_usage_daily_totals"."event_type" ASC NULLS FIRST, "billable_usage_daily_totals"."feature_qualified_name" ASC NULLS FIRST, "billable_usage_daily_totals"."operation_type" ASC NULLS FIRST

A whole year is read because every export covers the licence term.

Plan from Database Lab with a year of totals (730 rows): https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/61279/commands/166322

It reads the period through index_billable_usage_daily_totals_on_unique_tuple and sorts the 730 rows in memory, since the index orders NULLS LAST and the export orders NULLS FIRST.

How to set up and validate locally

Steps and expected output

Export records come from the daily totals

Seed per-namespace rows, refresh the totals, then build the records for the period. The output shown is from an actual run in a rolled-back transaction.

ns = Utilization::BillableUsage::DailyNamespaceAggregate
svc = Utilization::BillableUsage::Exports::UsageRecordsService
range = Date.new(2026, 8, 1)..Date.new(2026, 8, 31)

[[101, Date.new(2026, 8, 10), 40], [202, Date.new(2026, 8, 10), 60], [101, Date.new(2026, 8, 11), 5]].each do |root, date, qty|
  ns.create!(usage_date: date, root_namespace_id: root, event_type: 'secrets_read',
    feature_qualified_name: 'secrets_read', unit_of_measure: 'request', quantity: qty, events_count: qty)
end

Utilization::BillableUsage::RefreshDailyTotalsService.new.execute
records = svc.new(range).execute

records.size # => 2
records.first
# => {event_id: "f178638f-f516-5944-962a-28da0e229873", event_type: "secrets_read",
#     timestamp: "2026-08-10T12:00:00Z", quantity: 100.0, unit_of_measure: "request",
#     metadata: {feature_qualified_name: "secrets_read", schema_version: 1, events_count: 100,
#                quantity_kind: "counter"}}

svc.new(Date.new(2026, 8, 10)..Date.new(2026, 8, 10)).execute.first[:event_id] == records.first[:event_id] # => true

The extracted identity resolves from live instance state

Gitlab::BillingEvents::InstanceIdentity.to_h
# => {instance_id: "865a4e5c-...", unique_instance_id: "df73e4e6-...", host_name: "127.0.0.1",
#     instance_version: "19.5.0", realm: "SM", deployment_type: "self-managed"}

Each field should equal its source: Gitlab::GlobalAnonymousId.instance_id, Gitlab::GlobalAnonymousId.instance_uuid (unless a Cloud Connector token supplies a uid), Gitlab.config.gitlab.host, Gitlab.version_info.to_s, and CloudConnector.gitlab_realm mapped through REALM_MAP, with CloudConnector.deployment_type.

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Vijay Hawoldar

Merge request reports