Combined ref pipelines (source+target branch)

Problem to Solve

Teams need to keep master "green". It's a huge pain when someone merges a MR that breaks master because they didn't know of some conflict and their MR's pipeline was green. It's great that master has its own pipeline with tests so it won't accidentally deploy a broken codebase, but it's not nice that master is left broken, and any MR based on master will now fail as well.

If we had a way to build the combined result of the merge before merging, it would go a long way towards being more confident in the result.

Solution

In order to keep master green, we can run a pipeline on the state that master would be in after the merge - by combining the source and target branch into a new ref, then running a pipeline on that ref, we can better test that the combined result is also valid (at least moreso than if we test only the source branch.) To achieve this, this issue implements a rule that merge request pipelines now always run by default on the merged result: where the source and target branches are combined into a new ref and a pipeline is run that validates the result prior to merging.

There are some cases where creating a combined ref is not possible or not wanted; in these scenarios (for example, a source branch that has conflicts with the target, or a MR that is still in WIP status) we fallback to a "detached" state and run on the source branch ref - just like it works prior to this issue. Being in this detached state serves as a warning that you are working in a situation where you may eventually have merge problems, and helps highlight that you should get out of WIP status or resolve your merge conflicts as soon as reasonable.

Terminology

Pipelines for merge requests (or merge request pipelines) are pipelines that are created for merge requests. Of these there are two total types:

Attached merge request pipelines run on the merge ref. They by definition have one or more only: merge_requests jobs in the .gitlab-ci.yml which will be run for this pipeline. We're going to implement that as new behavior in this in this issue.
Detached merge request pipelines run on source branch with no combined merge attempted - this is the already in place behavior. It will occur when there is no merge request at all, if the merge request is in WIP, or if there are no only: merge_requests jobs in the .gitlab-ci.yml.

For clarity, a post-merge pipeline is just the regular pipeline that runs in the target branch after the MR is merged.

Additional Notes

The additional items here are important for implementing this complex feature, but are not required to understand how it will work.

This feature is an optional behavior that can be turned on at the project level. It is off by default for merge request pipelines until we have better contention handling (see next bullet)
If, after the merge request pipeline succeeds, the target branch has moved forward, then the ref is no longer valid and must be retried. In busy repos this can become a problem because you're nearly guaranteed that the target branch will have moved ahead. For how we're solving this problem in subsequent iterations, see https://gitlab.com/gitlab-org/gitlab-ee/issues/9186 and https://gitlab.com/gitlab-org/gitlab-ce/issues/57581. We are not going to try to auto-cancel these pipelines that will be unable to merge because there is still valuable information in knowing if the pipeline would have failed for other reasons.
If, after the merge request pipeline succeeds, the source branch has moved forward, the merge will not go through but the result will still be available to view.
We will now run at most one MR pipeline: either merge-result or detached. This is a change from previous behavior where we would run both.
If there is a merge conflict, auto-merging is not possible without human intervention (just the same as a normal merge.)
Merge request pipelines will run in the context of the ref; this means that only: target-branch type rule sets in the .gitlab-ci.yml will not be engaged. This is the safer behavior because if there are, for example, destructive actions performed via a production deployment on only: master then these will not be run. For complex behaviors, the implementer will need to be thoughtful about how they write their only/except rules to get the behavior they are looking for.
only/except: merge_requests (https://gitlab.com/gitlab-org/gitlab-ce/issues/15310) will continue to work exactly as they do today - if a MR is created, then only: merge_requests jobs will run.
Forking/cross-repo workflows are not supported in the MVC. See https://gitlab.com/gitlab-org/gitlab-ee/issues/9713 for how we will add these later. This would expand the scope considerably because of security reasons. We do though take it into account for the current implementation as to avoid having to rewrite this feature in the future in order to support that.
This feature is not available for fast forward merges yet. Follow https://gitlab.com/gitlab-org/gitlab-ce/issues/58226 for updates.

Behavior in pseudo-code:

if MR {
  if !WIP && !conflicts {
    run on merge result ref (merge pipeline - only: merge_request jobs run)
  } else {
    run on branch ref (detached merge pipeline - only: merge_request jobs run)
  }
} else {
  run normal pipeline (not merge pipeline - only: merge_request jobs NOT run)
}

This approach achieves compatibility with current workflows, keeps things as simple as possible, and gives the end user clear control over when to take advantage of this merge request pipeline through use of the WIP status.

User flow

The added functionality of this feature will come into play both when the user pushes new commits and wants to merge.

Commit flow

graph LR

UC>User pushes new commit]
MR{Branch part of MR?}
RBP[Run branch pipeline]
COW{WIP or are there conflicts?}
RMRP[Run detached merge request pipeline]
RMRRP[Run merge request pipeline]
PFSU["Pipeline feedback shown to user (mr pipeline feedback has priority)"]

UC-->MR
MR-->|Yes|COW
MR-->|No|RBP
MR-->|Yes|RBP
COW-->|Yes|RMRP
COW-->|No|RMRRP
RBP-->PFSU
RMRP-->PFSU
RMRRP-->PFSU

Merge flow

graph LR

UM>User wants to merge]
COW{WIP or are there conflicts?}
MUA[Merge button unavailable]
MA[Merge button available]
UPM[User presses merge button]
PTB[Pipeline for target branch is run]
PF{Pipeline failed?}
PS{Pipeline passed?}
PSR{pipeline still running?}
MWPS[Merge when pipeline succeeds button available]
AMP[Starts actual merge into target branch]
HMA{Has source or target advanced?}
PA[Merge process aborted]

UM-->COW
COW-->|Yes|MUA
COW-->|No|PF
PF-->|Yes|MUA
PF-->|No|PSR
PSR-->|Yes|MWPS
PSR-->|No|MA
MWPS-->UPM
UPM-->PS
PS-->|Yes|HMA
PS-->|No|PA
HMA-->|Yes|PA
HMA-->|No|AMP


MA-->UPM
AMP-->PTB

Design and design requirements

FE design specs!

💎 specs

Merge request widget

Relies on the same logic that would be implemented with https://gitlab.com/gitlab-org/gitlab-ce/issues/40246

Requirements	Mockups
Merge request is merge-able	We will show merge widget as normal (no changes)
Merge request is not merge-able	We will show merge widget as normal (no changes)
Merge request is being merged	Merge button has spinner and states `Merging` (no changes)
Fork merge request Warning (note: different from mockup) `Fork merge requests do not create merge request pipelines which validate a post merge result` Warning merge button (Merge when pipeline succeeds will also be warning colored as we rely on a normal branch pipeline in those cases)
Invalid merge request as target-branch advanced (linear history) merge button disabled warning: (note: different from mockup) `The target branch has advanced invalidating the merge request pipeline. Please update the source branch and retry merging`
Invalid merge request pipeline as target-branch advanced (non-linear history) merge when pipeline succeeds button (which triggers new pipeline when pressed) warning: (note: different from mockup) `The target branch has advanced invalidating the merge request pipeline. Please update the source branch and retry merging`
Merge request pipeline failed ( Note: decision to have no changes for now) merge when pipeline succeeds button (which triggers new pipeline when pressed) warning pipeline failed, retry merge or fix failure
Merge request is converted from WIP to non-WIP	Merge request pipeline is automatically triggered merge when pipeline succeeds button is shown (no changes)

Merge request widget pipeline block

Requirements	Mockups
Normal branch pipeline Unchanged
Detached merge request pipeline Ref now shows merge request ID/Link before the source branch
Merged result merge request pipeline Ref now shows merge request ID/Link before source branch and adds target branch

Pipeline list view

Requirements	Mockups
Normal branch pipeline Ref unchanged
Detached merge request pipeline Ref now shows merge request ID/Link merge request tag is removed `Detached` tag is added
Merged result merge request pipeline Ref now shows merge request ID/Link merge request tag is removed

Pipeline detail view widget

Requirements	Mockups
Normal branch pipeline Copy `from` changed to `for`
Detached merge request pipeline Ref now shows merge request ID/Link before the source branch merge request tag is removed `Detached` tag is added Copy `from` changed to `for`
Merged result merge request pipeline Ref now shows merge request ID/Link before source branch and adds target branch merge request tag is removed Copy `from` changed to `for`

Job detail view sidebar

Requirements	Mockups
Normal branch pipeline Copy `from` changed to `for`
Detached merge request pipeline Ref now shows merge request ID/Link before the source branch Copy `from` changed to `for`
Merged result merge request pipeline Ref now shows merge request ID/Link before source branch and adds target branch Copy `from` changed to `for`

Merge request list view

Requirements	Mockups
Pipeline status will include normal pipeline statuses, merge request pipelines statusses, merge request pipeline statuses, and post merge pipeline statusses, which ever one is the latest and thus most important

Project settings merge request settings

Requirements	Mockups
Added option for enabling merge request pipelines for merged result Restructured content (FE only, as same checkboxes still apply)

What happens on the backend?

To achieve this, the following steps occur in order:

User clicks merge and a merge request pipeline is initialized
The backend requests Gitaly to create new merge request reference already containing the merged codebase (see https://gitlab.com/gitlab-org/gitlab-ce/issues/47110)
GitLab refers to the new, merged .gitlab-ci.yml in the ref to generate the pipeline
Gitlab passes the ref and generated pipeline to the runner to execute (see also gitlab-runner#3983 (closed))
If the pipeline succeeds, and the target branch has not moved ahead, then the source branch is merged into the target.

Alternative dismissed approaches

The original MVC here would create a new ref in git that contains the merged codebase we can run against. This approach was discarded due to limitations for handling other common git workflows, such as forking. A branch-only approach is not "upgradable" to forking at a later date, and would require rewriting the feature.

A second proposal was discussed having the runner be responsible for the branch, but due to the possibility that the .gitlab-ci.yml can change in a branch, this does not work - the pipeline is processed by the rails application prior to the runner receiving it.

Usage Measure

We should measure how many people have the feature turned on via the usage ping.

Dogfooding

We (GitLab CE/EE) cannot use this feature in our large projects (i.e., gitlab-ce/gitlab-ee) yet for the following reasons:

We cannot accept MRs from community contributors because forked projects are not supported.
We often have over 40-50 merges in a day. Given our pipeline takes 1 and half hours, it likely causes competing merges and drop the 99% of merge request pipelines.

In order to adopt it, we need at least the follow up item https://gitlab.com/gitlab-org/gitlab-ce/issues/57581, but likely also https://gitlab.com/gitlab-org/gitlab-ee/issues/9186.

Backward compatibility

By the consequence of modifying Runner and Gitaly for supporting merge ref, users have to update their runners and gitaly to the new version as well as they update rails(CE/EE). If users forgot to update their runner, they will get a message in a pipeline job page that "'Your runner is outdated, please upgrade your runner'" (This is a generic message when runner is too old to support the feature). We already have this mechanizm today thus we don't need additional implementation.

Links / references

Marge-bot: https://smarketshq.com/marge-bot-for-gitlab-keeps-master-always-green-6070e9d248df

Tasks

Common

The functionality is available on EEP~EEU only. i.e. feature_available?(:merge_pipelines)
The functionality is built behind the feature flag merge_request_pipelines e.g. Feature.enabled?(:merge_pipelines) https://docs.gitlab.com/ee/development/rolling_out_changes_using_feature_flags.html#when-to-use-feature-flags
"BE:" means GitLab-rails, "BE-Runner" means GitLab-runner and "BE-Gitaly" means Gitaly.

Data to be persisted

sha = sha(merge-commit)
before_sha = merge_request.diff_base_sha
source_sha = sha(source-branch) = merge_request.diff_head_sha
target_sha = sha(target-branch)

Dataflow

User clicks a "MWPS" button with "Merge request pipelines" enabled.
GitLab-Rails askes Gitaly to create a merge ref (i.e. Executes MergeRequests::MergeToRefService).
GitLab-Rails creates a pipeline (i.e. Executes Ci::CreatePipelineService.execute(:merge_request) with ref: /merge_requests/:iid/merge).
GitLab-Runner requests a job.
GitLab-Rails returns the job's spec.
GitLab-Runner checkout the merge commit from given merge ref.
GitLab-Runner process a job.
GitLab-Runner sends job's results to GitLab-Rails.
Once the merge request pipeline succeeded, MergeRequests::MergeWhenPipelineSucceedsService#trigger is called (via PipelineSuccessWorker).

Outscoped tasks

Usage ping
- BE: Collect the count of projects which enabled "Run merge request pipelines" option

Edited Apr 05, 2019 by Jason Yavorska