Skip to content

Update Direct Transfer to support import in batches

George Koltsov requested to merge georgekoltsov/bulk-import-batch-support into master

What does this MR do and why?

This MR updates Direct Transfer and iterates over !124085 (merged) to start utilizing batched export on import side & import group/project relations in batches, if source instance supports batched export (16.2 onwards).

It introduces a number of new workers in order to achieve this:

  1. PipelineBatchWorker - responsible for importing a batch of group/project relation (e.g. 1000 issues of a project)
  2. FinishBatchedPipelineWorker - responsible for updating parent pipeline_tracker object after all batches are complete in order to progress with the import
  3. Updates PipelineWorker to enqueue batch jobs if source export is batched (it is from 16.2 onwards)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After
Migration doesn't use batch_trackers and executes entire relations within a single job Migration uses batch_trackers and executes relations import in batches
batched_migration_disabled.mov batched_migration_enabled.mov

How to set up and validate locally

  1. Have a group and project structure on .com (.com already supports batched export)
  2. Ger personal access token from .com and initiate a new import from .com to local gdk
  3. Enable :bulk_imports_batched_import_export feature flag
  4. Start a new direct transfer from .com to gdk
POST http://gdk.test:3000/api/v4/bulk_imports
PRIVATE-TOKEN: TOKEN
Content-Type: application/json

{
  "configuration": {
    "url": "https://gitlab.com/",
    "access_token": "TOKEN"
  },
  "entities": [
    {
      "source_full_path": "georgekoltsov-group",
      "source_type": "group_entity",
      "destination_slug": "georgekoltsov-group",
      "destination_namespace": "",
      "migrate_projects": true
    }
  ]
}
  1. Verify migration is successful
  2. Disable :bulk_imports_batched_import_export feature flag and start a new migration
  3. Verify migration is successful and there are no entries in bulk_import_batch_trackers table

Mentions #391224 (closed)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by George Koltsov

Merge request reports