Add a table recording generated usage exports

Vijay Hawoldarrequested to merge
18555-billable-usage-exports into master

What does this MR do and why?

Adds billable_usage_export_logs, one row per usage export an air-gapped instance generates: the period it covered, when it was generated and by whom, how many records it carried, and the payload checksum.

The file is handed to a customer, who uploads it to the Customers Portal by hand. Nothing logs that transfer, so this row and the record CustomersDot keeps on upload are the only two accounts of what happened. Matching them is what settles a disputed upload: the checksum shows a file came from this instance and was not edited and re-sealed, since whoever alters it recomputes the digest in the file but cannot alter the row; record_count shows whether records went missing on the way in; and generated_at answers whether an export happened at all, and for which period.

It is a fingerprint, not an archive. The payload is not stored, so questions about a quantity are answered from the daily usage tables.

Nothing on the instance reads this table back. Every export covers the whole subscription from licence start and CustomersDot deduplicates what it receives, so there is no bookmark to advance and no past export to decide anything from. The row is written before the file is served, so a build that errors cannot record an export that never left.

The table has no writer in this MR. The export endpoint in !257491 (merged) writes a row before it serves each file, and carries this MR's commit until it merges, so this MR must merge before !257491 (merged).

References

Database

One new table, no change to any existing table, no backfill, no data migration. gitlab_main_cell_local, table_size: small.

One row per export an admin generates, on air-gapped instances only, so realistically a handful per subscription per year.

CREATE TABLE billable_usage_export_logs (
    id bigint NOT NULL,
    created_at timestamp with time zone NOT NULL,
    updated_at timestamp with time zone NOT NULL,
    generated_at timestamp with time zone NOT NULL,
    generated_by_id bigint,
    period_start date NOT NULL,
    period_end date NOT NULL,
    record_count integer DEFAULT 0 NOT NULL,
    payload_checksum text NOT NULL,
    CONSTRAINT check_7214279a93 CHECK ((char_length(payload_checksum) <= 64)),
    CONSTRAINT check_billable_usage_export_logs_payload_checksum_format CHECK ((payload_checksum ~ '^[0-9a-f]{64}$'::text)),
    CONSTRAINT check_billable_usage_export_logs_period_end_after_start CHECK ((period_end >= period_start)),
    CONSTRAINT check_billable_usage_export_logs_record_count_non_negative CHECK ((record_count >= 0))
);

CREATE INDEX index_billable_usage_export_logs_on_generated_at
    ON billable_usage_export_logs USING btree (generated_at);
CREATE INDEX index_billable_usage_export_logs_on_generated_by_id
    ON billable_usage_export_logs USING btree (generated_by_id);
CREATE INDEX index_billable_usage_export_logs_on_payload_checksum
    ON billable_usage_export_logs USING btree (payload_checksum);

Two indexes for the two expected reads: the most recent export, for a line on the admin page reading "You last exported on ...", and a lookup by checksum when reconciling an upload. A third on generated_by_id serves the loose foreign key's cleanup. No unique index on the period, because the same period can legitimately be exported many times.

generated_by_id has a loose foreign key to users with async_nullify (config/gitlab_loose_foreign_keys.yml): when an admin is deleted, the export row stays and only generated_by_id is cleared.

payload_checksum is enforced in the database as well as the model: a CHECK requires 64 lowercase hex characters, the same format the model validates.

db/docs/data_retention/billable_usage_export_logs.yml declares indefinite_retention: each row is the instance's audit record of a file that left it, used to reconcile an upload, so rows are not aged out. The table stays at a handful of rows per instance per year.

Expected size and access

  • Growth. One row per export an admin generates, and only on offline-licensed instances: the endpoint that writes it (!257491 (merged)) is gated on Gitlab::BillingEvents::Client#local_persistence_enabled?, which excludes GitLab.com. Assuming an admin exports about monthly, that is around 3, 6 and 12 rows per instance at 3 months, 6 months and 1 year. On GitLab.com it stays empty.
  • Reads and writes. Writes are one insert per export served, so a few per month per instance. Rows are never updated. Nothing reads the table yet; the two expected reads are the most recent export, for a "You last exported on ..." line, and a lookup by checksum when reconciling an upload, both served by the two indexes.
  • Availability risk. None expected, for GitLab.com or self-managed: the table is write-once, tiny, and off every hot path.

Development seed

ee/db/fixtures/development/99_billable_usage_export_logs.rb seeds two rows, one generated by an admin and one with no user, so run-dev-fixtures-ee finds the new table populated and the nullable generated_by_id holds both a value and a null.

Migration output

db:migrate:

main: == 20260922060000 CreateBillableUsageExportLogs: migrating ====================
main: -- create_table(:billable_usage_export_logs)
main:    -> 0.0478s
main: == 20260922060000 CreateBillableUsageExportLogs: migrated (0.0579s) ===========

db:rollback:

main: == 20260922060000 CreateBillableUsageExportLogs: reverting ====================
main: -- drop_table(:billable_usage_export_logs)
main:    -> 0.0371s
main: == 20260922060000 CreateBillableUsageExportLogs: reverted (0.0510s) ===========

The migration uses change, so rolling back drops the table. That is safe while nothing writes to it; once !257491 (merged) ships, a rollback would discard the export records.

How to set up and validate locally

The table has no writer in this MR (the export endpoint in !257491 (merged) writes it), so validation is the model and the schema, in rails console. Output below is from an actual run, and the last step cleans up.

e = Utilization::BillableUsage::ExportLog

row = e.create!(period_start: Date.new(2026, 8, 1), period_end: Date.new(2026, 8, 31),
  generated_at: Time.utc(2026, 9, 1, 9, 30), generated_by_id: User.first&.id,
  record_count: 2, payload_checksum: SecureRandom.hex(32))
# => persisted

e.order(generated_at: :desc).first.generated_at.iso8601
# => "2026-09-01T09:30:00Z"    the read behind "You last exported on ..."

The model rejects the three things worth rejecting:

e.new(payload_checksum: 'not-a-digest', period_start: Date.new(2026, 8, 1),
      period_end: Date.new(2026, 8, 31), generated_at: Time.current, record_count: 1).valid?
# => false, payload_checksum ["is invalid"]

e.new(period_start: Date.new(2026, 8, 31), period_end: Date.new(2026, 8, 1),
      generated_at: Time.current, record_count: 1, payload_checksum: SecureRandom.hex(32)).valid?
# => false, period_end ["must be on or after period_start"]

e.new(record_count: -1, period_start: Date.new(2026, 8, 1), period_end: Date.new(2026, 8, 31),
      generated_at: Time.current, payload_checksum: SecureRandom.hex(32)).valid?
# => false, record_count ["must be greater than or equal to 0"]

And the database rejects them again when the model is bypassed, which is the point of having both:

row.update_column(:record_count, -1)
# ActiveRecord::StatementInvalid: check_billable_usage_export_logs_record_count_non_negative

row.update_column(:period_end, Date.new(2026, 7, 1))
# ActiveRecord::StatementInvalid: check_billable_usage_export_logs_period_end_after_start

row.update_column(:payload_checksum, 'Z' * 64)
# ActiveRecord::StatementInvalid: check_billable_usage_export_logs_payload_checksum_format

e.delete_all
bundle exec rspec ee/spec/models/utilization/billable_usage/export_log_spec.rb

21 examples, covering the same ground plus the presence validations and the checksum format at both sides of its boundary.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Vijay Hawoldar

Merge request reports