Create "Future of CI Pipeline Processing" design doc

What does this MR do and why?

Preview: https://main-ee-120867.docs.gitlab-review.app/ee/architecture/blueprints/ci_pipeline_processing/index.html

In GitLab CI, we have some problems with the current architecture and behavior of the pipeline processing. These problems confuse users and make it hard to understand the pipeline processing, lead to unexpected and complex behaviors, and make it hard to implement new features. In this design doc, we will discuss the problems and define the goals of the pipeline processing.

We will not discuss proposals in this MR.

Most of these problems have been discussed before in &6788.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

added Architecture Evolution Blueprint docs-only documentation grouppipeline authoring typemaintenance labels

assigned to @furkanayhan

added devopsverify sectionops labels

changed the description

	1 Warning
	Please add a merge request subtype to this merge request.

	2 Messages
	This merge request requires a review from an Architecture Evolution Coach.
	This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

Architecture Evolution Review

This merge request requires a review from an Architecture Evolution Coach.

Following files, that may require the additional review, have been changed:

• doc/architecture/blueprints/ci_pipeline_processing/index.md

Documentation review

The following files require a review from a technical writer:

• doc/architecture/blueprints/ci_pipeline_processing/index.md (Link to current live version)

The review does not need to block merging this merge request. See the:

• Metadata for the *.md files that you've changed. The first few lines of each *.md file identify the stage and group most closely associated with your docs change.
• The Technical Writer assigned for that stage and group.
• Documentation workflows for information on when to assign a merge request for review.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

changed the description

added 1 commit

• 49d669c4 - fix some typos

Compare with previous version

Setting label(s) Category:Pipeline Composition based on grouppipeline authoring.

added Category:Pipeline Composition label

mentioned in merge request !118638 (closed)

added 1 commit

• 0ecfaf9b - Add proposal

Compare with previous version

changed the description

changed the description

added 1 commit

• 6195aec3 - Refine goals

Compare with previous version

@ayufan Could you review this?

requested review from @ayufan

mentioned in issue #393129 (closed)

resolved all threads

added 3482 commits

• 6195aec3...59970b7c - 3476 commits from branch master
• 16123749 - Create "Future of CI Pipeline Processing" blueprint
• a1ffb9d8 - fix some typos
• 744fd6d4 - Add proposal
• 4a750b15 - Refine goals
• 41d3b61d - Change coach and approvers
• e81c919a - Small updates

Compare with previous version

marked this merge request as ready

marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

changed milestone to %16.2

Thanks @furkanayhan

I overall like your take on this problem definition. I commented on each aspect, as in some cases it is not fully clear what is the problem. A lot of things that is described as a problem seems to describe current state which after looking at them seems logically correct.

Unsure what was your intent whether this was to give a full picture of all behaviors relevant to the space, or outline all that are considered problem and needs to be fixed.

removed review request for @ayufan

TODO: what should we do with this scenario?

• https://gitlab.slack.com/archives/C0SFP840G/p1686221340648519
• https://gitlab.slack.com/archives/C0SFP840G/p1683730873434599?thread_ts=1683730867.298459&cid=C0SFP840G

build:
  stage: build
  script: sleep 10
  allow_failure: true

test:
  stage: test
  script: exit 0

• If build is canceled, test runs.
• If build is failed, test runs.

mentioned in issue #348146 (closed)

requested review from @ayufan

added 1 commit

• 25a39206 - small changes

Compare with previous version

added sectionci label and removed sectionops label

Should we also reconsider dependencies? It looks so confusing with needs.

We can even create a pipeline like this;

build:
  script: exit 0
  needs: [test]

test:
  script: exit 0
  dependencies: [build]

Let's see if we explored #388866 in this MR.

mentioned in issue #388866

added 3009 commits

• 25a39206...8aff4bf8 - 3001 commits from branch master
• c2e843cb - Create "Future of CI Pipeline Processing" blueprint
• a3460151 - fix some typos
• c965f731 - Add proposal
• a69d67a7 - Refine goals
• bbfdd67b - Change coach and approvers
• 457c96a7 - Small updates
• 3e738d04 - small changes
• f2e11947 - some improvements

Compare with previous version

added 1 commit

• 4c5fff82 - some improvements

Compare with previous version