Skip to content

Use Pipelines for Merge Requests with change detection

Rémy Coutable requested to merge use-merge-request-pipeline-with-change-detection into master

CE port (not needed anymore): gitlab-foss!32524 (closed)

What does this MR do?

This implements Pipelines for Merge Requests with change detection.

The default only, which every job should use (except master-only or tags-only jobs) is defined as follows:

.default-only:
  only:
    refs:
      - master
      - /-stable/
      - /^d+-d+-auto-deploy-d+$/
      - merge_requests
      - tags

Then each job can define when to run depending on the changes:

Only run when code-related files are changed:

.only-code-changes:
  only:
    changes:
      - ".gitlab/ci/**/*"
      - ".{eslintignore,gitattributes,nvmrc,prettierrc,stylelintrc,yamllint}"
      - ".{codeclimate,eslintrc,gitlab-ci,haml-lint,haml-lint_todo,rubocop,rubocop_todo,scss-lint}.yml"
      - ".csscomb.json"
      - "Dangerfile"
      - "Dockerfile.assets"
      - "*_VERSION"
      - "Gemfile{,.lock}"
      - "Rakefile"
      - "{babel.config,jest.config}.js"
      - "config.ru"
      - "{package.json,yarn.lock}"
      - "{app,bin,config,danger,db,ee,fixtures,haml_lint,lib,public,rubocop,scripts,spec,symbol,vendor}/**/*"
      - "doc/README.md"  # Some RSpec test rely on this file

Only run when QA-related files are changed:

.only-qa-changes:
  only:
    changes:
      - ".dockerignore"
      - "qa/**/*"

Note: These jobs will only run when QA changes are made. Most of the jobs should run when code-related or QA-related changes are made, see .only-code-qa-changes below.

Only run when docs are changed:

.only-docs-changes:
  only:
    changes:
      - ".gitlab/route-map.yml"
      - "doc/**/*"
      - ".markdownlint.json"

This will allow to enable the Pipelines for Merged Results feature, and then the Merge Trains feature.

Only run when code-related or QA-related files are changed:

.only-code-qa-changes:
  only:
    changes:
      - ".gitlab/ci/**/*"
      - ".{eslintignore,gitattributes,nvmrc,prettierrc,stylelintrc,yamllint}"
      - ".{codeclimate,eslintrc,gitlab-ci,haml-lint,haml-lint_todo,rubocop,rubocop_todo,scss-lint}.yml"
      - ".csscomb.json"
      - "Dangerfile"
      - "Dockerfile.assets"
      - "*_VERSION"
      - "Gemfile{,.lock}"
      - "Rakefile"
      - "{babel.config,jest.config}.js"
      - "config.ru"
      - "{package.json,yarn.lock}"
      - "{app,bin,config,danger,db,ee,fixtures,haml_lint,lib,public,rubocop,scripts,spec,symbol,vendor}/**/*"
      - "doc/README.md"  # Some RSpec test rely on this file
      - ".dockerignore"
      - "qa/**/*"

Note: This is basically the concatenation of .only-code-changes and .only-qa-changes, since we cannot extend multiple only.changes (the last would win).

Verification matrix

Code changes

Project Reference pipeline Test MR Missing jobs Extra jobs After pipeline status
EE canonical https://gitlab.com/gitlab-org/gitlab/pipelines/80241102 !15761 (merged) ["docs lint", "ee-files-location-check", "ee-specific-lines-check", "migration:upgrade-pg-ce-to-ee", "review-docs-cleanup", "review-docs-deploy-manual"] ["review-cleanup-failed-deployment"] ✅
EE fork https://gitlab.com/leetickett/gitlab-ee/pipelines/80210286 !16129 (merged) ["docs lint", "ee_compat_check", "migration:upgrade-pg-ce-to-ee", "parallel-spec-reports"] [] ✅
EE dev https://dev.gitlab.org/gitlab/gitlab-ee/pipelines/125594 https://dev.gitlab.org/gitlab/gitlab-ee/merge_requests/1259 ["docs lint", "migration:upgrade-pg-ce-to-ee", "parallel-spec-reports"] [] ✅

Notes:

  • Missing docs lint, review-docs-cleanup is an improvement as this job is now only created when there are Docs changes.
  • Missing review-docs-deploy-manual has been renamed review-docs-deploy since it's now only created when there are Docs changes.
  • Missing parallel-spec-reports is an improvement as this job is now only created when there are Code or QA changes and when namespace is gitlab-org (since it collects results from automated QA against a Review App, which is only available for canonical pipelines).
  • Missing ee_compat_check for the EE fork is an improvement as this job made no sense for EE anyway (and the job wasn't doing anything in that case).
  • Missign ee-files-location-check, ee-specific-lines-check, migration:upgrade-pg-ce-to-ee were removed
  • Extra review-cleanup-failed-deployment is a new job that was added in the meantime.
  • Extra no_ee_check is an improvement as there was no reason not to have it for forks or dev.

QA changes

Project Reference pipeline Test MR Missing jobs Extra jobs After pipeline status
EE canonical https://gitlab.com/gitlab-org/gitlab/pipelines/80455813 !16017 (closed) ["code_quality", "dependency_scanning", "ee-files-location-check", "ee-specific-lines-check", "karma", "lint-ci-gitlab", "memory-static", "qa-frontend-node:10", "qa-frontend-node:8", "qa-frontend-node:latest", "sast"] ["review-cleanup-failed-deployment"] 🔵
EE fork https://gitlab.com/rymai/gitlab-ee/pipelines/80268787 !16018 (closed) ["code_quality", "dependency_scanning", "ee_compat_check", "karma", "lint-ci-gitlab", "memory-static", "parallel-spec-reports", "qa-frontend-node:10", "qa-frontend-node:8", "qa-frontend-node:latest", "sast"] [] ✅
EE dev https://dev.gitlab.org/gitlab/gitlab-ee/pipelines/125595 https://dev.gitlab.org/gitlab/gitlab-ee/merge_requests/1260 ["code_quality", "karma", "lint-ci-gitlab", "memory-static", "parallel-spec-reports", "qa-frontend-node:10", "qa-frontend-node:8", "qa-frontend-node:latest"] [] ✅

Notes:

  • Missing code_quality is an improvement as this job is now only created when there are Code changes.
  • Missing compile-assets pull-cache is an improvement as this job is now only created when there are Code changes.
  • Missing dependency_scanning is an improvement as this job is now only created when there are Code changes.
  • Missing karma is an improvement as this job is now only created when there are Code changes.
  • Missing lint-ci-gitlab is an improvement as this job is now only created when there are **/*.yml changes.
  • Missing memory-static is an improvement as this job is now only created when there are Code changes.
  • Missing qa-frontend-node:* is an improvement as these jobs are now only created when there are Code changes.
  • Missing sast is an improvement as this job is now only created when there are Code changes.
  • Missign ee-files-location-check, ee-specific-lines-check were removed
  • Extra review-cleanup-failed-deployment is a new job that was added in the meantime.

Docs changes

Project Reference pipeline Test MR Missing jobs Extra jobs After pipeline status
EE canonical https://gitlab.com/gitlab-org/gitlab/pipelines/80459631 !16015 (closed) ["ee-files-location-check", "ee-specific-lines-check"] [] ✅
EE fork https://gitlab.com/rymai/gitlab-ee/pipelines/80270205 !16016 (closed) ["lint-ci-gitlab"] [] ✅
EE dev https://dev.gitlab.org/gitlab/gitlab-ee/pipelines/125596 https://dev.gitlab.org/gitlab/gitlab-ee/merge_requests/1261 [] [] ✅

Notes:

  • Missing lint-ci-gitlab is an improvement as this job is now only created when there are **/*.yml changes.
  • Missign ee-files-location-check, ee-specific-lines-check were removed

Specific branches

Project Reference pipeline Test MR Missing jobs Extra jobs After pipeline status
Geo branch (code change) https://gitlab.com/gitlab-org/gitlab/pipelines/80518212 !16184 (merged) ["docs lint", "ee-files-location-check", "ee-specific-lines-check", "migration:upgrade-pg-ce-to-ee", "quick-rspec geo pg ee", "quick-rspec geo pg-10 ee", "review-docs-cleanup", "review-docs-deploy-manual"] ["quick-rspec geo pg ee 1/3", "quick-rspec geo pg ee 2/3", "quick-rspec geo pg ee 3/3", "quick-rspec geo pg-10 ee 1/3", "quick-rspec geo pg-10 ee 2/3", "quick-rspec geo pg-10 ee 3/3", "review-cleanup-failed-deployment"] ✅

Notes:

  • Missing docs lint, review-docs-cleanup is an improvement as this job is now only created when there are Docs changes.
  • Missing review-docs-deploy-manual has been renamed review-docs-deploy since it's now only created when there are Docs changes.
  • Missing quick-rspec geo pg ee is an improvement as this job is now parallelized.
  • Missign ee-files-location-check, ee-specific-lines-check, migration:upgrade-pg-ce-to-ee were removed
  • Extra review-cleanup-failed-deployment is a new job that was added in the meantime.

Conformity

Closes #27749 (closed), #31642 (closed).

Edited by Rémy Coutable

Merge request reports