Add billable usage daily aggregate table for air-gapped instances
What does this MR do and why?
Air-gapped self-managed instances have no outbound connectivity, so the billable usage events that normally stream to CustomersDot never leave the instance, and those customers cannot participate in usage-based billing.
This adds the on-instance store their exports will read from. Writing to it is handled in !249324 (merged).
One row per (usage_date, event_type, feature_qualified_name, root_namespace_id, operation_type),
so a day's events for one feature and scope fold into a single row.
Refs: https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/18550
Changelog: other EE: true
Columns worth explaining
event_aggregate_uuididentifies the aggregate rather than the row, so it cannot be the primary key: it has to be unique across every instance, because CustomersDot dedupes uploads on this value alone. It is derived deterministically from the unique tuple, so re-exporting a period is recognised as the same record rather than as new usage.root_namespace_idis the scope.secrets_storedemits one snapshot per root namespace and a snapshot replaces rather than adds, so without it those writes would overwrite each other and the day's figure would be whichever job ran last. Nullable, because not every producer has a namespace to report.operation_typeis recorded as reported. CustomersDot decides which types it charges for by checkingNON_BILLABLE_OPERATION_TYPES, so billable and non-billable operations have to be separate rows. The instance never interprets this value.schema_versionmakes a future change to the aggregate format additive rather than a backfill.NULLS NOT DISTINCTon the unique index, so a null scope or operation type still collides on upsert instead of inserting a fresh row on every write.- No foreign key on
root_namespace_id, and an entry inspec/db/schema_spec.rb'signored_fk_columns_map: billing rows must outlive namespace deletion, so a cascading FK would destroy data that has not been exported yet. Cell-local tables must not declare a sharding key, sodb/docshas none. - No
metadatacolumn. Product confirmed on &22758 that air-gapped DAP pricing is flat and token-independent, and asked that token counts not be collected at all. That left nothing which is both populated on an air-gapped instance and read downstream.
Migration up
$ bundle exec rails db:migrate:up:main VERSION=20260804000000
main: == 20260804000000 CreateBillableUsageDailyAggregates: migrating ===============
main: -- create_table(:billable_usage_daily_aggregates)
main: -- quote_column_name(:event_type)
main: -> 0.0000s
main: -- quote_column_name(:unit_of_measure)
main: -> 0.0000s
main: -- quote_column_name(:feature_qualified_name)
main: -> 0.0000s
main: -- quote_column_name(:operation_type)
main: -> 0.0000s
main: -> 0.0234s
main: == 20260804000000 CreateBillableUsageDailyAggregates: migrated (0.0282s) ======Migration down
$ bundle exec rails db:migrate:down:main VERSION=20260804000000
main: == 20260804000000 CreateBillableUsageDailyAggregates: reverting ===============
main: -- drop_table(:billable_usage_daily_aggregates)
main: -> 0.0269s
main: == 20260804000000 CreateBillableUsageDailyAggregates: reverted (0.0335s) ======How to set up and validate locally
-
Run the migration:
bundle exec rails db:migrate:up:main VERSION=20260804000000 -
In
rails console, exercise the clauses the writer will use. Rows are written by upsert against the unique tuple, so repeated events fold into one row. The writer itself is in !249324 (merged); these are its clauses.K = Utilization::BillableUsage::DailyAggregate UNIQUE_BY = %i[usage_date event_type feature_qualified_name root_namespace_id operation_type] DATE = Date.new(2026, 8, 10) def uuid_for(event_type, fqn, root_namespace_id, operation_type) Digest::UUID.uuid_v5( Gitlab::GlobalAnonymousId.instance_uuid, [DATE, event_type, fqn, root_namespace_id, operation_type].join(':') ) end COUNTER = Arel.sql( 'quantity = billable_usage_daily_aggregates.quantity + EXCLUDED.quantity, ' \ 'events_count = billable_usage_daily_aggregates.events_count + EXCLUDED.events_count, ' \ 'updated_at = EXCLUDED.updated_at' ) GAUGE = Arel.sql( 'quantity = EXCLUDED.quantity, ' \ 'events_count = billable_usage_daily_aggregates.events_count + EXCLUDED.events_count, ' \ 'updated_at = EXCLUDED.updated_at' ) def row(event_type, fqn, unit, quantity, root_namespace_id = nil, operation_type = nil) { usage_date: DATE, event_type: event_type, feature_qualified_name: fqn, root_namespace_id: root_namespace_id, operation_type: operation_type, unit_of_measure: unit, quantity: quantity, events_count: 1, event_aggregate_uuid: uuid_for(event_type, fqn, root_namespace_id, operation_type) } end def upsert(attrs, clause) K.upsert_all([attrs], unique_by: UNIQUE_BY, record_timestamps: true, on_duplicate: clause) end # counter: three secrets_read events for one root namespace 3.times { upsert(row('secrets_read', 'secrets_read', 'request', 1, 7), COUNTER) } K.find_by!(event_type: 'secrets_read', root_namespace_id: 7).slice(:quantity, :events_count) # gauge: two snapshots for the same scope replace rather than add upsert(row('secrets_stored', 'secrets_stored', 'secret', 1150, 7), GAUGE) upsert(row('secrets_stored', 'secrets_stored', 'secret', 1162, 7), GAUGE) # a second scope is its own row, not an overwrite upsert(row('secrets_stored', 'secrets_stored', 'secret', 40, 9), GAUGE) # operation_type keeps billable and non-billable operations apart 4.times { upsert(row('duo_agent_platform_workflow_completion', 'software_development/v1', 'request', 1, 7, 'regular'), COUNTER) } upsert(row('duo_agent_platform_workflow_completion', 'software_development/v1', 'request', 1, 7, 'compaction_auto'), COUNTER)Run as written, this gives:
case result counter ×3 quantity=3,events_count=3gauge ×2, same scope quantity=1162,events_count=2two scopes 2 rows, 1162and40two operation types 2 rows, 4and1, with distinctevent_aggregate_uuid
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.