Skip to content

Add issues importer

Alexandru Croitor requested to merge add-issues-importer into master

What does this MR do?

Adds implementation of the issues importer workers and services.

Import flow

Import is based on several import stages: StartImportWorker, ImportLabelsWorker, ImportIssuesWorker, ImportAttachmentsWorker, ImportNotesWorker and FinishImportWorker

This MR touches only ImportIssuesWorker and partly FinishImportWorker stages. Other stages are empty and just pass the work to next stage without doing any actual work.

Flow Steps

  1. Import is triggered by setting project.import_state to scheduled state, with a force-import value set in project.import_data. We need the force-import flag to differentiate between jira import vs mirror sync, that uses same project.import_state object.
  2. When project.import_state it will schedule a StartImportWorker that will actually start the import
  3. StartImportWorker schedules the ImportLabelsWorker
  4. ImportLabelsWorker after importing labels(its empty at this point) schedules ImportIssuesWorker
  5. ImportIssuesWorker calls jira api and pulls a batch of issues, batch size is set to 1000. For each issue a ImportIssuesWorker is scheduled that actually saves data to DB.
  6. If there are more than 1000 issues to be imported the worker will not schedule next stage, but rather schedule the ImportIssuesWorker again after first batch of 1000 ImportIssuesWorker workers finish.
  7. When ImportIssuesWorker reaches last page it will pass on to next stage and schedule the ImportAttachmentsWorker
  8. ImportAttachmentsWorker -> ImportNotesWorker
  9. ImportNotesWorker -> FinishImportWorker
  10. FinishImportWorker sets the project.import_state to finished and does some cleanup as for instance removing force-import flag so that if mirror sync triggers a sync it will not perceive it as an import.

Important

  • All workers are behind jira_issue_import feature flag, which is disabled by default.

re #208910 (closed)

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Alexandru Croitor

Merge request reports