Add possibility to import all gists

What does this MR do and why?

This is the third part of the implementation for GitHub Importer: API for bulk import of personal gists to personal snippets

From the original issue Problem to solve:

Problem to solve

We can import repos from GitHub, but for a more complete migration away from GitHub, we would need to have a way to import GitHub Gists as well.

With the release of Snippets with multiple Files, it is now possible to migrate GitHub Gists (up to 10 files) directly into GitLab Snippets. As we haven't had any requests to increase that limit in the product and allow snippets to have more files, we won't try to make that possible with import either.

All workers and importers are added under gitlab/github_gists_import namespace since gitlab/github_import is all about project import.

The overall feature implementation is split into 3 parts that need to be merged in sequential order:

1. ✅ Merged MR1 adds the ImportGistWorker that performs the import of one GitHub gist into a GitLab snippet and notifies JobWaiter upon importing. It calls GistImporter that is responsible to create a Snippet from gist object represented by Representation::Gist. It returns an error when the imported gist has more than 10 files.
2. ✅ Merged MR2 - Add importer to fetch gists from Github and schedule worker for each gist to be imported adds GistsImporter to fetch all gists from GitHub and schedules workers (from MR1) for each gist to be imported.
3. This MR3 adds possibility to import all gists. Import::Github::GistsImportService is responsible for scheduling StartImportWorker based on the import status that is stored in Redis and is kept for 24 hours. This worker executes GistsImporter (from MR2) to fetch all gists from GitHub and schedules worker (from MR1) for each gist to be imported. FinishImportWorker completes the import process by setting the status as finished.
4. MR4 - Add import all gists endpoint adds new public API endpoint and documentation for it.

Diagram

graph TD
1(Start) -->|Send request| 3[POST /import/github/gists]
3 --> 4[GistsImportService]
subgraph MergeRequest3
4 -->|Check status| 6{status == 'started'}
6 -->|Yes| 7[Return 422 error]
6 -->|No| 8[StartImportWorker]
subgraph MergeRequest2
  8 -->|Log and execute importer| 9[GistsImporter]
  9 --> A[Create Waiter]
  A -->|Fetch Gists from Github| B[GithubImport::Client]
  B -->|Schedule Bulk Perform| C[ImportGistWorker]
  subgraph MergeRequest1
  C -->|Log and execute importer| D[GistImporter]
  D -->|Check execution result| E{Success?}
  E -->|Yes| F[Notify Waiter]
  E -->|No| G[Log Error]
  G --> F
  D -->|check if more than 10 files| H{valid?}
  H -->|Yes| I[Create Snippet]
  H -->|No| J[Return Error]
  end
end
9 -->|Check execution result| 10{Success?}
10 -->|Yes| 11[FinishImportWorker]
11 -->|Check Waiter| 12{jobs_remaining = 0}
12 -->|Yes| 13[Set Status to 'finished']
13 --> 14(End)
12 -->|No| 11
10 -->|No| 15{Rate limit reached?}
15 -->|Yes| 8
15 -->|No| 16[Log and raise error]
end

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

gists_import_service1

How to set up and validate locally

From rails console:

1. Import::Github::GistsImportService.new(<user>, github_access_token: <github_access_token>).execute
2. Observe Snippets Dashboard - imported items should appear

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.