Add issues importer
What does this MR do?
Adds implementation of the issues importer workers and services.
Import flow
Import is based on several import stages: StartImportWorker, ImportLabelsWorker, ImportIssuesWorker, ImportAttachmentsWorker, ImportNotesWorker and FinishImportWorker
This MR touches only ImportIssuesWorker and partly FinishImportWorker stages. Other stages are empty and just pass the work to next stage without doing any actual work.
Flow Steps
- Import is triggered by setting project.import_state to
scheduledstate, with aforce-importvalue set inproject.import_data. We need theforce-importflag to differentiate between jira import vs mirror sync, that uses sameproject.import_stateobject. - When
project.import_stateit will schedule aStartImportWorkerthat will actually start the import -
StartImportWorkerschedules theImportLabelsWorker -
ImportLabelsWorkerafter importing labels(its empty at this point) schedulesImportIssuesWorker -
ImportIssuesWorkercalls jira api and pulls a batch of issues, batch size is set to 1000. For each issue aImportIssuesWorkeris scheduled that actually saves data to DB. - If there are more than 1000 issues to be imported the worker will not schedule next stage, but rather schedule the
ImportIssuesWorkeragain after first batch of 1000ImportIssuesWorkerworkers finish. - When
ImportIssuesWorkerreaches last page it will pass on to next stage and schedule theImportAttachmentsWorker -
ImportAttachmentsWorker->ImportNotesWorker -
ImportNotesWorker->FinishImportWorker -
FinishImportWorkersets theproject.import_statetofinishedand does some cleanup as for instance removingforce-importflag so that if mirror sync triggers a sync it will not perceive it as an import.
Important
- All workers are behind
jira_issue_importfeature flag, which is disabled by default.
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec -
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by Alexandru Croitor