Skip to content

Improve performance while exporting CI pipelines

Related to: #391593 (closed)

What does this MR do and why?

CI pipeline statuses can't be preloaded correctly because statuses is a polymorphic association, and each "real" association requires different associations to be preloaded.

For instance, CommitStatus is the base class, and the child classes are Ci::Build, Ci::Bridge, and GenericCommitStatus. And GenericCommitStatus doesn't have the associations metadata, which prevents us from preloading this association with the statuses

This change updates Import/Export to export the "real" associations instead of the polymorphic association statuses.

See the example below of how the ci_pipelines were exported before and after:

ci_pipelines.ndjson format before

{
  "ref": "ref",
  "sha": "602c025ab40150333aa06fa8971446c057156e09",
  ...
  "stages": [
    {
      "project_id": 1,
      ...
      "statuses": [
         {
            "type": "Ci::Build"
         },
         {
            "type": "Ci::Bridges"
         },
         {
            "type": "GenericCommitStatus"
         }
      ]
   }
}

ci_pipelines.ndjson format after

{
  "ref": "ref",
  "sha": "602c025ab40150333aa06fa8971446c057156e09",
  ...
  "stages": [
    {
      "project_id": 1,
      ...
      "builds": [
         {
         }
      ],
      "bridges": [
         {
         }
      ],
      "generic_commit_statuses": [
         {
         }
      ]
   }
}

And to still import old export files that use the statuses association, the settings import_only_tree was added to the import_export.yml. The settings let the import process consider the statuses association as valid, while during the export process, the settings is ignored.

This change also updates BulkImport aka GitLab migration, to export and import the new associations.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

It's necessary to perform 4 tests

Import/Export

  • Import an export file with the old association statuses and check if the CI Pipelines were imported correctly
  • Import an export file with the new associations builds, bridges, and generic_commit_statuses and check if the CI Pipelines were imported correctly

GitLab Migration

  • Import an export file with the old association statuses and check if the CI Pipelines were imported correctly
  • Import an export file with the new associations builds, bridges, and generic_commit_statuses and check if the CI Pipelines were imported correctly

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports