Report commit status based on deterministic pipelines groups
Description
We recently changed the rules for calculating commit status in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7333.
Previously we used compound status from all pipelines associated with given SHA and branch. This means that if someone had multiple pipelines where the most recent one was successful but the previous are failed we would report failure.
This approach was considered as a bug by most of the people, but the real problem here is that pipelines for given SHA / reference pair are not deterministic. This happens because one can modify pipeline behavior using only: - triggers
and some other mechanisms that are available.
This leads to the problem where someone may depend on this situation, and run multiple pipelines on the same commit / branch, and still depend on the compound status.
See discussion in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7333 and https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7333#note_20060402, main issue is https://gitlab.com/gitlab-org/gitlab-ce/issues/20560, and https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6801#note_17468470.
We can have multiple pipelines for SHA/branch pair. Pipeline gets created when someone pushes the code. It can also be created using the UI (Run pipeline button). It can also be created using legacy builds triggers API (that will be replaced by new pipeline API). When someone uses API, he will be able to provide environment variables, what can effectively change the status of the pipeline, because pipelines are not deterministic for given SHA/ref pair.
Added confusion
The confusion over the implementation of only: merge-requests
added to the urgency of this issue as without #55011 (closed) it is counter-intuitive that the "merge request" pipeline is smaller/a sub-set of the "branch" pipeline in many cases. This causes MWPS
or maintainers to be confused about if this MR is truly tested.
Part of this issue should be to ensure all pipelines on a ref / HEAD of a branch that's going to be merged are green before allowing merge / MWPS
Proposal
Report commit status basing on pipelines grouped by their determinism. We should pick the most recent pipeline from each deterministic group and use one most recent pipeline from each group to calculate compound status.
We still need to devise mechanism for deterministic pipelines groups.