Add IID preallocator for project import/export
What does this MR do and why?
dds an IID preallocator for project import/export to prevent IID collisions during import.
- Adds an import_export_preallocate_iids feature flag to gate all changes
- Adds Gitlab::ImportExport::Project::MaxIidsSaver which writes a max_iids.json file to the export archive containing the maximum IID for each tracked resource (issues, MRs, milestones, pipelines, designs)
- Adds Gitlab::Import::IidPreallocator which reads max_iids.json at the start of import and claims IIDs via AtomicInternalId before any relations are restored, closing the race window where concurrent resource creation could collide with imported IIDs
- Hooks MaxIidsSaver into both ParallelExportService (UI/API parallel export path) and ExportService (rake task/legacy path), gated behind the feature flag
- Calls IidPreallocator in Importer#execute immediately after the archive is extracted and version-checked, before any restorers (including repo/wiki restore) run — this ensures IIDs are claimed as early as possible in the import process, minimising the race window between project creation and IID reservation
- Fixes reset_counters_and_iids to also flush namespace-scoped issue InternalId records — issue IIDs are scoped to project_namespace (not project), so the existing InternalId.flush_records!(project: self) was not clearing them. Adds a targeted InternalId.where(namespace_id: project_namespace_id, usage: :issues).delete_all to handle this
- Removes the redundant InternalId.flush_records!(namespace: project.project_namespace) from RelationImportWorker#perform_post_import_tasks since reset_counters_and_iids now handles it
- EE extensions add support for iterations (via MaxIidsSaver) and epics/iterations (via IidPreallocator)
References
Screenshots or screen recordings
| Before | After |
|---|---|
![]() |
![]() |
![]() |
|
![]() |
|
![]() |
How to set up and validate locally
-
Set up a source project set up with a number of resources: 7 issues, 4 MRs, 7 pipelines, 6 milestones, etc.
-
Enable the feature flag in rails console:
Feature.enable(:import_export_preallocate_iids) -
Export this project via Settings > General > Export project
-
Extract the archive and verify
max_iids.jsonis present at the root with correct max IIDs -
To check
InternalIds are correct, addsleep 20to line 2756 ofapp/models/project.rb
2752 # The import assigns iid values on its own, e.g. by re-using GitHub ids.
2753 # Flush existing InternalId records for this project for consistency reasons.
2754 # Those records are going to be recreated with the next normal creation
2755 # of a model instance (e.g. an Issue).
2756 + sleep 30
2757 InternalId.flush_records!(project: self)
2758 InternalId.flush_records!(namespace: project_namespace, usage: :issues)
2759 update_project_counter_caches
- Initiate an import
- In the rails console, verify
InternalIdrecords are pre-created:
project = Project.last
InternalId
.where(project_id: project.id)
.or(InternalId.where(namespace_id: project.project_namespace_id))
.pluck(:usage, :last_value)
You should see rows for issues, merge_requests, milestones, ci_pipelines, and design_management_designs with last_value matching the max IIDs from the source project.
=> [["issues", 7], ["merge_requests", 5], ["ci_pipelines", 7], ["design_management_designs", 11], ["milestones", 6]]
- Remove the previous
sleepand addsleep 10toGroup::RelationTreeRestorer#process_relation!to delay each resource import so you can manually create your own resources while the project is importing (this tests the race condition is fixed):
lib/gitlab/import_export/group/relation_tree_restorer.rb
92 def process_relation!(relation_key, relation_definition, extra_track_scope: {})
93 + sleep 10
94 @relation_reader.consume_relation(@importable_path, relation_key).each do |data_hash, relation_index|
95 with_progress_tracking(**progress_tracking_options(relation_key, relation_index, extra_track_scope)) do
96 process_relation_item!(relation_key, relation_definition, relation_index, data_hash)
97 end
...
- Initiate a new import.
- While the import is still running, create new resources manually.
- Observe that the new resources are created with an IID of the
max_iid + 1e.g. if themax_iidfor issues is7, the new issue should have an IID of8 - For sanity remove the
sleep 10, disable the feature flag, restart the gdk, and check that an import completes as normal.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Relates to Allocate IIDs on Project Import/Export (#520997 - closed)
Closes Issues created while a project being imported b... (#519457 - closed)




