Conditionally include templates to speed up GitLab pipelines

Problem

Merge request pipelines for gitlab-org/gitlab project hit timeout.

Context

We spend a lot of time in evaluating job rules and persisting the pipeline. For P99 metrics we spend up to ~27 seconds in evaluating job rules and up to ~30 seconds persisting the pipeline. (source)

Improvements in persisting the pipeline is something we need to address internally. However, the job rules evaluation could be greatly improved by moving more and more rules in the include statement whenever possible.

Example 1

In the .gitlab-ci.yml we include all local templates if we are not in the security project. This rule is not granular enough which makes us rely on each job to have their own rules.

If we take an example of file included like .gitlab/ci/yaml.gitlab-ci.yml, we see that each of the 3 jobs extends either .yaml-line:rules or .lint-pipeline-yaml:rules or lint-metrics-yaml:rules each being evaluated at job level.

One improvement here could be to use a more specific rule when including it in the .gitlab-ci.yml:

include:
  - local: .gitlab/ci/yaml.gitlab-ci.yml
    rules:
      - <<: *if-merge-request-security-canonical-sync
        when: never
      - changes: ["**/*.{yml,yaml}{,.*}"] # if no changes to any YAML files we don't even add the 3 jobs

Example 2

Instead of always including .gitlab/ci/docs.gitlab-ci.yml , which adds 9 jobs before we filter them by job-level rules, we could include it conditionally:

include:
  - local: .gitlab/ci/yaml.gitlab-ci.yml
    rules:
      - <<: *if-merge-request-security-canonical-sync
        when: never
      - <<: *if-dot-com-gitlab-org-merge-request
        changes: *docs-patterns