Add billable usage daily aggregate table for air-gapped instances

What does this MR do and why?

Air-gapped self-managed instances have no outbound connectivity, so the billable usage events that normally stream to CustomersDot never leave the instance, and those customers cannot participate in usage-based billing.

This adds the on-instance store their exports will read from. Writing to it is handled in !249324 (merged).

One row per (usage_date, event_type, feature_qualified_name, root_namespace_id, operation_type), so a day's events for one feature and scope fold into a single row.

Refs: https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/18550

Changelog: other EE: true

Columns worth explaining

  • event_aggregate_uuid identifies the aggregate rather than the row, so it cannot be the primary key: it has to be unique across every instance, because CustomersDot dedupes uploads on this value alone. It is derived deterministically from the unique tuple, so re-exporting a period is recognised as the same record rather than as new usage.
  • root_namespace_id is the scope. secrets_stored emits one snapshot per root namespace and a snapshot replaces rather than adds, so without it those writes would overwrite each other and the day's figure would be whichever job ran last. Nullable, because not every producer has a namespace to report.
  • operation_type is recorded as reported. CustomersDot decides which types it charges for by checking NON_BILLABLE_OPERATION_TYPES, so billable and non-billable operations have to be separate rows. The instance never interprets this value.
  • schema_version makes a future change to the aggregate format additive rather than a backfill.
  • NULLS NOT DISTINCT on the unique index, so a null scope or operation type still collides on upsert instead of inserting a fresh row on every write.
  • No foreign key on root_namespace_id, and an entry in spec/db/schema_spec.rb's ignored_fk_columns_map: billing rows must outlive namespace deletion, so a cascading FK would destroy data that has not been exported yet. Cell-local tables must not declare a sharding key, so db/docs has none.
  • No metadata column. Product confirmed on &22758 that air-gapped DAP pricing is flat and token-independent, and asked that token counts not be collected at all. That left nothing which is both populated on an air-gapped instance and read downstream.

Migration up

$ bundle exec rails db:migrate:up:main VERSION=20260804000000

main: == 20260804000000 CreateBillableUsageDailyAggregates: migrating ===============
main: -- create_table(:billable_usage_daily_aggregates)
main: -- quote_column_name(:event_type)
main:    -> 0.0000s
main: -- quote_column_name(:unit_of_measure)
main:    -> 0.0000s
main: -- quote_column_name(:feature_qualified_name)
main:    -> 0.0000s
main: -- quote_column_name(:operation_type)
main:    -> 0.0000s
main:    -> 0.0234s
main: == 20260804000000 CreateBillableUsageDailyAggregates: migrated (0.0282s) ======

Migration down

$ bundle exec rails db:migrate:down:main VERSION=20260804000000

main: == 20260804000000 CreateBillableUsageDailyAggregates: reverting ===============
main: -- drop_table(:billable_usage_daily_aggregates)
main:    -> 0.0269s
main: == 20260804000000 CreateBillableUsageDailyAggregates: reverted (0.0335s) ======

How to set up and validate locally

  1. Run the migration:

    bundle exec rails db:migrate:up:main VERSION=20260804000000
  2. In rails console, exercise the clauses the writer will use. Rows are written by upsert against the unique tuple, so repeated events fold into one row. The writer itself is in !249324 (merged); these are its clauses.

    K = Utilization::BillableUsage::DailyAggregate
    UNIQUE_BY = %i[usage_date event_type feature_qualified_name root_namespace_id operation_type]
    DATE = Date.new(2026, 8, 10)
    
    def uuid_for(event_type, fqn, root_namespace_id, operation_type)
      Digest::UUID.uuid_v5(
        Gitlab::GlobalAnonymousId.instance_uuid,
        [DATE, event_type, fqn, root_namespace_id, operation_type].join(':')
      )
    end
    
    COUNTER = Arel.sql(
      'quantity = billable_usage_daily_aggregates.quantity + EXCLUDED.quantity, ' \
      'events_count = billable_usage_daily_aggregates.events_count + EXCLUDED.events_count, ' \
      'updated_at = EXCLUDED.updated_at'
    )
    
    GAUGE = Arel.sql(
      'quantity = EXCLUDED.quantity, ' \
      'events_count = billable_usage_daily_aggregates.events_count + EXCLUDED.events_count, ' \
      'updated_at = EXCLUDED.updated_at'
    )
    
    def row(event_type, fqn, unit, quantity, root_namespace_id = nil, operation_type = nil)
      {
        usage_date: DATE, event_type: event_type, feature_qualified_name: fqn,
        root_namespace_id: root_namespace_id, operation_type: operation_type,
        unit_of_measure: unit, quantity: quantity, events_count: 1,
        event_aggregate_uuid: uuid_for(event_type, fqn, root_namespace_id, operation_type)
      }
    end
    
    def upsert(attrs, clause)
      K.upsert_all([attrs], unique_by: UNIQUE_BY, record_timestamps: true, on_duplicate: clause)
    end
    
    # counter: three secrets_read events for one root namespace
    3.times { upsert(row('secrets_read', 'secrets_read', 'request', 1, 7), COUNTER) }
    K.find_by!(event_type: 'secrets_read', root_namespace_id: 7).slice(:quantity, :events_count)
    
    # gauge: two snapshots for the same scope replace rather than add
    upsert(row('secrets_stored', 'secrets_stored', 'secret', 1150, 7), GAUGE)
    upsert(row('secrets_stored', 'secrets_stored', 'secret', 1162, 7), GAUGE)
    
    # a second scope is its own row, not an overwrite
    upsert(row('secrets_stored', 'secrets_stored', 'secret', 40, 9), GAUGE)
    
    # operation_type keeps billable and non-billable operations apart
    4.times { upsert(row('duo_agent_platform_workflow_completion', 'software_development/v1', 'request', 1, 7, 'regular'), COUNTER) }
    upsert(row('duo_agent_platform_workflow_completion', 'software_development/v1', 'request', 1, 7, 'compaction_auto'), COUNTER)

    Run as written, this gives:

    case result
    counter ×3 quantity=3, events_count=3
    gauge ×2, same scope quantity=1162, events_count=2
    two scopes 2 rows, 1162 and 40
    two operation types 2 rows, 4 and 1, with distinct event_aggregate_uuid

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Vijay Hawoldar

Merge request reports

Loading
Loading