Backfill organization_id on bulk_import_exports via sharding-key trigger
What does this MR do and why?
Part 2 of 2: Backfill organization_id on bulk_import_exports via sharding-key trigger
Populates the organization_id column added in Part 1 (!241088 (merged)):
- Adds a
BEFORE INSERT OR UPDATEtrigger derivingorganization_idfrom the grandparent (projectwhenproject_idis set, elsenamespaceviagroup_id). Also covers new rows on insert. - Queues the
BackfillBulkImportExportsOrganizationIdBBM to fire the trigger across existing rows. - Updates
db/docs/bulk_import_exports.ymland addsbulk_import_exportstoallowed_organization_id_violationsfor the in-progress nullable state.
NOT NULL, FK validation, and promoting organization_id into the active sharding_key block come in a later phase, after the backfill runs on production.
Note for reviewers: this is a stacked MR built on Part 1 (!241088 (merged)) — please review that one as well.
Migration details
db/post_migrate/20260617101510_add_sharding_key_trigger_on_bulk_import_exports.rb
db/post_migrate/20260617101511_queue_backfill_bulk_import_exports_organization_id.rb
lib/gitlab/background_migration/backfill_bulk_import_exports_organization_id.rbQuery plan
The organization_id column + index aren't on the database-lab clone yet (not deployed until Part 1 lands), so I added them on the clone (exec ALTER TABLE ... ADD COLUMN + CREATE INDEX) and ran EXPLAIN on the per-batch update.
Per-batch UPDATE plan
ModifyTable on public.bulk_import_exports (cost=0.43..209.29 rows=0 width=0) (actual time=536.685..536.686 rows=0 loops=1)
Buffers: shared hit=17215 read=974 dirtied=534 written=3
WAL: records=6025 fpi=528 bytes=4364226
-> Index Scan using bulk_import_exports_pkey on public.bulk_import_exports (cost=0.43..209.29 rows=866 width=14) (actual time=1.981..26.637 rows=856 loops=1)
Index Cond: ((bulk_import_exports.id >= 1) AND (bulk_import_exports.id <= 1000))
Filter: (bulk_import_exports.organization_id IS NULL)
Rows Removed by Filter: 0
Buffers: shared hit=740 read=24 dirtied=2
WAL: records=2 fpi=2 bytes=16278
Settings: jit = 'off', random_page_cost = '1.5', work_mem = '230MB', seq_page_cost = '4', effective_cache_size = '472585MB'Each batch scans by primary key (id range) with organization_id IS NULL as an in-scan filter — each_batch iterates over the PK, so it doesn't depend on idx_bulk_import_exports_on_organization_id.
MR acceptance checklist
- Migrations tested up and down locally
- BBM spec passing
- Database review
References
- Issue #600457 (closed), epic &21584 (closed)
- Part 1: !241088 (merged)
- Follows the sibling
bulk_import_trackersrollout.