Some reports need to run after docs MRs merge
Issue
After a discussion with @iamricecake about how Code Quality works, we looked at examples in gitlab
and realized that the intermittent behavior is likely caused by the pipelines that run after a merge, which sometimes (often?) don't run Code Quality. For example, when a docs MR merges, only a minimal pipeline runs on master
. This small docs master
pipeline does not run most CI/CD jobs that generate reports, like the Code Quality report.
When Code Quality runs in a code MR, it checks the latest pipeline in the target branch. If that target branch is master
(which is 99% of the time), it needs to have the Code Quality report artifact available to compare again. But if the latest pipeline was a docs merge, (or another pipeline that doesn't have Code Quality), there is no artifact to compare to, and the MR therefore thinks that all the violations in the MR are new, which makes the Code Quality reporting not useful to engineers. At times, it just seems to spin forever when it can't find the artifact (this might also be caused when the latest pipeline was skipped with [ci skip]
.
I only looked at Code Quality, but there is the possibility that this could improve the behavior of other reports in MRs. For now, we can start with code quality and see if that improves the behavior.
I suspect docs merges could be one of the bigger culprits, because we merge docs MRs more often than code MRs (I think) and the pipelines finish quickly, overriding the previous "latest" MR.
Related to gitlab-docs#1088 (moved)
Proposal
We should run Code quality
on all pipelines on the default branch, including docs MRs, so that there is always an artifact present to compare against in MRs.
-
Current rules:
.reports:rules:code_quality: rules: - if: '$CODE_QUALITY_DISABLED' when: never - <<: *if-default-refs changes: *code-backstage-patterns
-
Proposed fix:
.reports:rules:code_quality: rules: - if: '$CODE_QUALITY_DISABLED' when: never - <<: *if-default-refs changes: *code-backstage-patterns - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
Screenshots
Some of the behaviors we see:
-
Code quality does find an artifact (optimal case):
-
Code quality doesn't find the artifact (sub-optimal case, devs can't get useful information, unless it get highlighted in diff):
-
Code quality gets confused (sometimes the case where latest pipeline was skipped with
[ci-skip]
, maybe sometimes when it can't find the artifact, this spinner goes forever):