Backfill organization_id on bulk_import_exports via sharding-key trigger

What does this MR do and why?

Part 2 of 2: Backfill organization_id on bulk_import_exports via sharding-key trigger

Populates the organization_id column added in Part 1 (!241088 (merged)):

  • Adds a BEFORE INSERT OR UPDATE trigger deriving organization_id from the grandparent (project when project_id is set, else namespace via group_id). Also covers new rows on insert.
  • Queues the BackfillBulkImportExportsOrganizationId BBM to fire the trigger across existing rows.
  • Updates db/docs/bulk_import_exports.yml and adds bulk_import_exports to allowed_organization_id_violations for the in-progress nullable state.

NOT NULL, FK validation, and promoting organization_id into the active sharding_key block come in a later phase, after the backfill runs on production.

Note for reviewers: this is a stacked MR built on Part 1 (!241088 (merged)) — please review that one as well.

Migration details

db/post_migrate/20260617101510_add_sharding_key_trigger_on_bulk_import_exports.rb
db/post_migrate/20260617101511_queue_backfill_bulk_import_exports_organization_id.rb
lib/gitlab/background_migration/backfill_bulk_import_exports_organization_id.rb

Query plan

The organization_id column + index aren't on the database-lab clone yet (not deployed until Part 1 lands), so I added them on the clone (exec ALTER TABLE ... ADD COLUMN + CREATE INDEX) and ran EXPLAIN on the per-batch update.

Query plan

Per-batch UPDATE plan
 ModifyTable on public.bulk_import_exports  (cost=0.43..209.29 rows=0 width=0) (actual time=536.685..536.686 rows=0 loops=1)
   Buffers: shared hit=17215 read=974 dirtied=534 written=3
   WAL: records=6025 fpi=528 bytes=4364226
   ->  Index Scan using bulk_import_exports_pkey on public.bulk_import_exports  (cost=0.43..209.29 rows=866 width=14) (actual time=1.981..26.637 rows=856 loops=1)
         Index Cond: ((bulk_import_exports.id >= 1) AND (bulk_import_exports.id <= 1000))
         Filter: (bulk_import_exports.organization_id IS NULL)
         Rows Removed by Filter: 0
         Buffers: shared hit=740 read=24 dirtied=2
         WAL: records=2 fpi=2 bytes=16278
Settings: jit = 'off', random_page_cost = '1.5', work_mem = '230MB', seq_page_cost = '4', effective_cache_size = '472585MB'

Each batch scans by primary key (id range) with organization_id IS NULL as an in-scan filter — each_batch iterates over the PK, so it doesn't depend on idx_bulk_import_exports_on_organization_id.

MR acceptance checklist

  • Migrations tested up and down locally
  • BBM spec passing
  • Database review

References

Edited by Bojan Marjanovic

Merge request reports

Loading
Loading