CI: `stop_review` will not fire if pipeline hasn't succeeded
[Updated description]
Problem
stop_review
job in an MR pipeline not being triggered when MR is merged.
Doubts
- We reuse environment in different jobs in different stages of the pipeline in order to have the CI variables automatically populate with environment information. They are (in the order of position in pipeline)
- review (stage: review)
- stop_review (stage: review)
- review_specs (stage: specs)
- review_helm_test (stage: qa)
- debug_review (stage: qa, when: on_failure)
- How exactly is
stop_review
triggered? Is it like this: "When an MR is merged, thestop_review
job from the same stage as of the job which caused latest deployment to that environment is triggered"?- In our case, latest deployment need not be the
review
job, because we are reusing the environment in later stages, and those jobs (review_specs
andreview_helm_test
) are considered to be deployments by GitLab. So, in a completely green pipeline,review_helm_test
is the latest deployment to an environment.
- In our case, latest deployment need not be the
- Does the existing issue https://gitlab.com/gitlab-org/gitlab-ce/issues/45704, where sidebar shows different set of jobs, usually from another pipeline ran on the same commit have some relation to this?
- Is it just a UI level bug or is it an API level one?
- Is
stop_review
not triggered automatically for a branch, if it has been already run sometime in the past, maybe for a different commit.
Impact
- Deployments from branches are not removed, thus eating up resources and hence filling the cluster so new deployments can't happen
- Review apps are not being cleaned up, as noted in https://gitlab.com/gitlab-org/gitlab-ee/issues/6884
Related to https://gitlab.com/charts/helm.gitlab.io/issues/314
If a MR's pipeline did not pass all stages prior to stop_review
, then stop_review
will not fire automatically when the MR is merged. This results in the requirement to manually run the job (can be done no matter the state of the pipeline).
- We need to determine if it is possible to have this job run, despite failures, at the appropriate time.
- Even if the above can be done, as an interim measure, we should document that whomever performs the MR should ensure that
stop_review
will be fired, manually if necessary.
cc @twk3
Edited by Balasankar 'Balu' C