Add IID preallocator for project import/export

What does this MR do and why?

dds an IID preallocator for project import/export to prevent IID collisions during import.

  • Adds an import_export_preallocate_iids feature flag to gate all changes
  • Adds Gitlab::ImportExport::Project::MaxIidsSaver which writes a max_iids.json file to the export archive containing the maximum IID for each tracked resource (issues, MRs, milestones, pipelines, designs)
  • Adds Gitlab::Import::IidPreallocator which reads max_iids.json at the start of import and claims IIDs via AtomicInternalId before any relations are restored, closing the race window where concurrent resource creation could collide with imported IIDs
  • Hooks MaxIidsSaver into both ParallelExportService (UI/API parallel export path) and ExportService (rake task/legacy path), gated behind the feature flag
  • Calls IidPreallocator in Importer#execute immediately after the archive is extracted and version-checked, before any restorers (including repo/wiki restore) run — this ensures IIDs are claimed as early as possible in the import process, minimising the race window between project creation and IID reservation
  • Fixes reset_counters_and_iids to also flush namespace-scoped issue InternalId records — issue IIDs are scoped to project_namespace (not project), so the existing InternalId.flush_records!(project: self) was not clearing them. Adds a targeted InternalId.where(namespace_id: project_namespace_id, usage: :issues).delete_all to handle this
  • Removes the redundant InternalId.flush_records!(namespace: project.project_namespace) from RelationImportWorker#perform_post_import_tasks since reset_counters_and_iids now handles it
  • EE extensions add support for iterations (via MaxIidsSaver) and epics/iterations (via IidPreallocator)

References

Screenshots or screen recordings

Before After
Before_-_No_max_iids_json After_-_max_iids_json
max-iids
new_issue
new_milestone

How to set up and validate locally

  1. Set up a source project set up with a number of resources: 7 issues, 4 MRs, 7 pipelines, 6 milestones, etc.

  2. Enable the feature flag in rails console:

    Feature.enable(:import_export_preallocate_iids)
  3. Export this project via Settings > General > Export project

  4. Extract the archive and verify max_iids.json is present at the root with correct max IIDs

  5. To check InternalIds are correct, add sleep 20 to line 2756 of app/models/project.rb

2752   # The import assigns iid values on its own, e.g. by re-using GitHub ids.
2753   # Flush existing InternalId records for this project for consistency reasons.
2754   # Those records are going to be recreated with the next normal creation
2755   # of a model instance (e.g. an Issue).
2756 + sleep 30
2757   InternalId.flush_records!(project: self)
2758   InternalId.flush_records!(namespace: project_namespace, usage: :issues)
2759   update_project_counter_caches
  1. Initiate an import
  2. In the rails console, verify InternalId records are pre-created:
project = Project.last

InternalId
  .where(project_id: project.id)
  .or(InternalId.where(namespace_id: project.project_namespace_id))
  .pluck(:usage, :last_value)

You should see rows for issues, merge_requests, milestones, ci_pipelines, and design_management_designs with last_value matching the max IIDs from the source project.

=> [["issues", 7], ["merge_requests", 5], ["ci_pipelines", 7], ["design_management_designs", 11], ["milestones", 6]]
  1. Remove the previous sleep and add sleep 10 to Group::RelationTreeRestorer#process_relation! to delay each resource import so you can manually create your own resources while the project is importing (this tests the race condition is fixed):
lib/gitlab/import_export/group/relation_tree_restorer.rb
92   def process_relation!(relation_key, relation_definition, extra_track_scope: {})
93 +    sleep 10      
94      @relation_reader.consume_relation(@importable_path, relation_key).each do |data_hash, relation_index|
95         with_progress_tracking(**progress_tracking_options(relation_key, relation_index, extra_track_scope)) do
96           process_relation_item!(relation_key, relation_definition, relation_index, data_hash)
97         end
         ...
  1. Initiate a new import.
  2. While the import is still running, create new resources manually.
  3. Observe that the new resources are created with an IID of the max_iid + 1 e.g. if the max_iid for issues is 7, the new issue should have an IID of 8
  4. For sanity remove the sleep 10, disable the feature flag, restart the gdk, and check that an import completes as normal.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Relates to Allocate IIDs on Project Import/Export (#520997 - closed)

Closes Issues created while a project being imported b... (#519457 - closed)

Edited by Carla Drago

Merge request reports

Loading