Add bulk import gists endpoint

What does this MR do and why?

The third part of the implementation GitHub Importer: API for bulk import of personal gists to personal snippets

From the original issue Problem to solve:

Problem to solve

We can import repos from GitHub, but for a more complete migration away from GitHub, we would need to have a way to import GitHub Gists as well.

With the release of Snippets with multiple Files, it is now possible to migrate GitHub Gists (up to 10 files) directly into GitLab Snippets. As we haven't had any requests to increase that limit in the product and allow snippets to have more files, we won't try to make that possible with import either.

All workers and importers are added under gitlab/github_gists_import namespace since gitlab/github_import is all about project import.

The overall feature implementation is split into 3 parts that need to be merged in sequential order:

1. ✅ Merged MR1 adds the ImportGistWorker that performs the import of one GitHub gist into a GitLab snippet and notifies JobWaiter upon importing. It calls GistImporter that is responsible to create a Snippet from gist object represented by Representation::Gist. It returns an error when the imported gist has more than 10 files.
2. ✅ Merged MR2 - Add importer to fetch gists from Github and schedule worker for each gist to be imported adds GistsImporter to fetch all gists from GitHub and schedules workers (from MR1) for each gist to be imported. FinishImportWorker completes the import process by setting the status as finished.
3. ✅ Merged MR3 - Add possibility to import all gists adds possibility to import all gists. Import::Github::GistsImportService is responsible for scheduling StartImportWorker based on the import status that is stored in Redis and is kept for 24 hours. This worker executes GistsImporter (from MR2) to fetch all gists from GitHub and schedules worker (from MR1) for each gist to be imported. FinishImportWorker completes the import process by setting the status as finished.
4. This MR4 adds new public API endpoint and documentation for it.

Diagram

graph TD
1(Start) -->|Send request| 3[POST /import/github/gists]
3 --> 4[GistsImportService]
subgraph MergeRequest3
4 -->|Check status| 6{status == 'started'}
6 -->|Yes| 7[Return 422 error]
6 -->|No| 8[StartImportWorker]
subgraph MergeRequest2
  8 -->|Log and execute importer| 9[GistsImporter]
  9 --> A[Create Waiter]
  A -->|Fetch Gists from Github| B[GithubImport::Client]
  B -->|Schedule Bulk Perform| C[ImportGistWorker]
  subgraph MergeRequest1
  C -->|Log and execute importer| D[GistImporter]
  D -->|Check execution result| E{Success?}
  E -->|Yes| F[Notify Waiter]
  E -->|No| G[Log Error]
  G --> F
  D -->|check if more than 10 files| H{valid?}
  H -->|Yes| I[Create Snippet]
  H -->|No| J[Return Error]
  end
end
9 -->|Check execution result| 10{Success?}
10 -->|Yes| 11[FinishImportWorker]
11 -->|Check Waiter| 12{jobs_remaining = 0}
12 -->|Yes| 13[Set Status to 'finished']
13 --> 14(End)
12 -->|No| 11
10 -->|No| 15{Rate limit reached?}
15 -->|Yes| 8
15 -->|No| 16[Log and raise error]
end

Screenshots or screen recordings

When import started it returns 202 Accepted status code. If import is triggered again during import process it returns error

Screen_Recording_2022-11-08_at_09.14.40

Secret Gists become Private Snippets:

Screen_Recording_2022-11-08_at_09.16.18

Public Gists become Public Snippets

Screen_Recording_2022-11-08_at_09.15.47

If the gist have more than 10 files import is skipped and an error is logged with the message:

"message":"importer failed","error.message":"exсeeds max file limit"

How to set up and validate locally

POST to {{host}}/api/v4/import/github/gists

Request body:

{ "personal_access_token": <github_personal_access_token> }

Observe Snippets Dashboard to include imported items.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.