Skip to content

RUN AS-IF-FOSS Add Group Import via GraphQL

George Koltsov requested to merge georgekoltsov/group-import-using-graphql into master

What does this MR do?

This MR introduces a new way to import Groups from another GitLab instance using GraphQL, so there is no need in dealing with exported tar.gz archive.

Key things introduced

  1. Group information is fetched using GraphQL. Done using graphlient gem (that uses GitHub's graphql-client gem under the hood). Main reason for not using graphql-client gem directly is being able to use dynamic queries and dynamic client, instead of static ones, as we need to initialize a new GraphQL client for each user that uses the migration tool https://github.com/ashkan18/graphlient#dynamic-vs-static-queries
  2. Fetched data processing is done using a new ETL Pipeline concept (see below)

Sequence diagram

image

⚙ Data processing

On a high level, ETL (https://en.wikipedia.org/wiki/Extract,_transform,_load) pipeline consists 3 main components: Extractors, Transformers and Loaders.

  • Extractor - extracts data from source (in our case, it's from GraphQL). Can be more than one extractor if we need to get information from different places (e.g. GraphQL extractor & HTTP extractor)
  • Transformer - as simple as possible class that typically has small responsibility of performing ideally one or several data transformations. In our case, we want to trasnform fetched data from GraphQL (e.g. remove GraphQL specific keys from response hash)
  • Loader - behaviour responsible for 'loading'/saving data. In this MR it's a simple class that calls Groups::CreateService

To illustrate, a GroupPipeline would look like this:

image

Approaching import as data processing ETL pipeline allows to have better visibility into data transformations (comparing to existing generic solution that can be hard to understand on what's going on without debugging into the import process) as well as easier extension and modification.

🔬 Testing

Make sure sidekiq is running

# Create groups

@created_groups = []

5.times do |i|
  g = Group.create!(name: "source group#{i}", path: "sgroup#{i}")
  g.add_owner(User.first)
  @created_groups << g
end

@destination_group = Group.create!(name: 'destination', path: 'destination')
@destination_group.add_owner(User.first)

# Import

importer_user = User.first
import_params = []

@created_groups.each do |group|
  import_params << {
    source_type: 'group_entity',
    source_name: group.name,
    source_full_path: group.full_path,
    destination_name: "IMPORTED #{group.name}",
    destination_namespace: @destination_group.path
  }
end

credentials = { url: 'http://127.0.0.1:3000', access_token: 'token' }

BulkImportService.new(importer_user, import_params, credentials).execute

image

This functionality just covers basic creation of Groups, without any other metadata. Next steps will be introducing Epics/Boards/Labels import (out of scope for this MR).

Out of scope of this MR

As we've just started working on this project, we try to iterate and keep changes smaller (is not really a case in this MR, sorry!). So to keep things smaller, there are a few things that are out of scope of this MR and are going to be handles as followups:

Conformity

Edited by George Koltsov

Merge request reports