Skip to content

Draft: POC - Execute GitHub Import stages in parallel

Rodrigo Tomonari requested to merge rodrigo/parallel-github-import into master

What does this MR do and why?

This is a proof-of-concept for optimizing GitHub Import performance by running stages in parallel.

Related to: #438280 (closed)

This change introduces a new way to trigger GitHub Import stages. Previously, stages were responsible for initiating the next stage. However, with this new approach, stages only need to change their status to "finish", and the StageOrchestratorWorker will determine which stage to run next.

The main benefit of this method is that we can easily modify the order in which stages are executed. For example, the StageOrchestratorWorker could use a class that executes stages in sequence, or a different class that runs stages in parallel. To keep things simple, the proof-of-concept implemented the stage execution logic in the StageOrchestratorWorker.

This change also subdivides some stages into smaller ones to increase the parallelism of stage execution.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Rodrigo Tomonari

Merge request reports