ci: Avoid blanket changes to avoid unexpected run
What does this MR do and why?
Avoid blanket changes to avoid unexpected run
Documentation to this caveat: https://docs.gitlab.com/ee/ci/yaml/#ruleschanges
You should use
rules: changes
only with branch pipelines or merge request pipelines. You can userules: changes
with other pipeline types, butrules: changes
always evaluates totrue
when there is no Gitpush
event. Tag pipelines, scheduled pipelines, manual pipelines, and so on do not have a Git push event associated with them. Arules: changes
job is always added to those pipelines if there is no if that limits the job to branch or merge request pipelines.
If we trigger a pipeline from another pipeline, regardless whatever the actual changes were, it'll always evaluate to true which can be unexpected.
If we also apply if-default-refs
it would at least limit the scope to those default scenarios we care most about.
That being said, adding if-default-refs
to those conditions will not stop the incident from happening, because we can see that if $FORCE_GITLAB_CI
is being set, it's also considered if-default-refs
, and the triggered pipeline did also contain $FORCE_GITLAB_CI
as we can see from gitaly!4986 (merged)
Therefore, I don't expect this to change the actual behaviours, nor does it prevent from cache poisoning.
However I think it's likely still beneficial to have this:
- More clear what to expect
- If we tweak
if-default-refs
we can tweak the behaviour when needed
Background
See gitlab-org/quality/engineering-productivity/master-broken-incidents#45 (comment 1159064802)
- An unmerged Gitaly merge request gitaly!4986 (merged) triggered a cross-project pipeline on GitLab
toon-gitaly-version
branch which also has an unmerged merge request: !102281 (merged) - !102281 (merged) showed a mixed of pipelines with its owned merge requests pipelines and "branch pipelines" triggered from Gitaly.
- Merge request pipelines did not run
update-gitaly-binaries-cache
norupdate-tests-metadata
because relevant files did not change. However, the "branch pipelines" did run those becausechanges
are alwaystrue
to those pipelines, because those were not pushed but triggered. This was discovered from !102727 (comment 1158917472)
Merge request pipeline without update-gitaly-binaries-cache
: https://gitlab.com/gitlab-org/gitlab/-/pipelines/678677901
Branch pipeline having update-gitaly-binaries-cache
: https://gitlab.com/gitlab-org/gitlab/-/pipelines/683127875
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
See gitlab-org/quality/engineering-productivity/master-broken-incidents#45 (comment 1159064802)
Merge request reports
Activity
assigned to @godfat-gitlab
changed milestone to %15.6
added maintenancepipelines label
added typemaintenance label
Pipeline Changes
This merge request contains changes to the pipeline configuration for the GitLab project.
Please consider the effect of the changes in this merge request on the following:
- Effects on different pipeline types
- Effects on non-canonical projects:
gitlab-foss
security
dev
- personal forks
- Effects on pipeline performance
Please consider communicating these changes to the broader team following the communication guideline for pipeline changes
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer maintenanceworkflow / maintenancepipelines for CI, Danger Jay Montal (
@jmontal
) (UTC-7, 7 hours behind@godfat-gitlab
)Kerri Miller (
@kerrizor
) (UTC-8, 8 hours behind@godfat-gitlab
)To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangeradded 1 commit
- 4433450e - Avoid blanket changes to avoid unexpected run
mentioned in merge request !102727 (merged)
- Resolved by Lin Jen-Shin
@ddieulivol Given that you have better context for this, what do you think?
requested review from @ddieulivol
marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed
- Resolved by Rémy Coutable
mentioned in issue #379128 (closed)
mentioned in merge request gitaly!4986 (merged)
requested review from @rymai and removed review request for @ddieulivol
- Resolved by Rémy Coutable
question (non-blocking): From the MR description:
we can see that if
$FORCE_GITLAB_CI
is being set, it's also consideredif-default-refs
I don't think I understand this statement.
I see in the code that if
FORCE_GITLAB_CI
is set, we'll always create a pipeline (probably a branch pipeline, which are the problematic ones in this discussion), BUT thanks to this MR, if the branch pipeline isn't from a stable ref (which is quite likely), then*if-default-refs
would befalse
and the job wouldn't be executed? Is this reasoning wrong?Also,
I don't expect this to change the actual behaviours, nor does it prevent from cache poisoning.
Probably related to what's above, but it should definitely help for cache poisoning, no?
Without this MR, if anybody triggers a branch pipeline from their branch, it would for instance update the gitaly binary cache if the gitaly version were to be different in the branch...At least that's my understanding of it
- Resolved by Rémy Coutable
question (non-blocking): Do you know what kind of pipelines are we generally running, and where can I see that information?
Given the Gitaly incident discussed in this MR, I could see we have branch pipelines and MR pipelines at the very least, but are we configuring those in the
workflow:
clause of the CI?I also checked https://docs.gitlab.com/ee/development/pipelines/internals.html, but we mention
pipelines
, without the type (probably because it would depend on the use-case...)For context: I'm asking this question because I initially thought about only having certain types of pipelines running to avoid having that
changes:
defaulting totrue
in some cases.
@ddieulivol
, thanks for approving this merge request.This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.
For more info, please refer to the following links:
- Resolved by Lin Jen-Shin
That being said, adding
if-default-refs
to those conditions will not stop the incident from happening, because we can see that if$FORCE_GITLAB_CI
is being set, it's also consideredif-default-refs
, and the triggered pipeline did also contain$FORCE_GITLAB_CI
as we can see from gitaly!4986 (merged)Isn't this the actual problem here? Setting
FORCE_GITLAB_CI
allows anyone with permission to trigger an arbitrary pipeline with variables. I think we should consider disallow that.For instance, in this case, the testing done in gitaly!4986 (merged) could have been done by modifying the
GITALY_SERVER_VERSION
file directly in agitlab-org/gitlab
MR instead of !102281 (merged) + gitaly!4986 (merged).
That being said, the change looks reasonable and can only improve things (and we can always remove
$FORCE_GITLAB_CI
fromif-default-refs
and/orworkflow:rules
later).
@godfat-gitlab Looks good to me, thanks!
added 520 commits
-
4433450e...7e899f82 - 519 commits from branch
master
- 3cbc827c - Merge branch 'master' into 'ci-avoid-blanket-changes'
-
4433450e...7e899f82 - 519 commits from branch
Thank you for resolving the conflicts for me! The failure shouldn't be relevant, but here's the failure log: https://gitlab.com/gitlab-org/gitlab/-/jobs/3284944401
2022-11-07 14:58:24 +0000 Rack app ("POST /namespace27/project63/-/issues" - (127.0.0.1)): #<Gitlab::QueryLimiting::Transaction::ThresholdExceededError: Too many SQL queries were executed in Projects::IssuesController#create: a maximum of 100 is allowed but 104 SQL queries were executed
Might be flaky from the default branch?
mentioned in commit 60b63f1e
added workflowstaging-canary label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
mentioned in issue #378981 (closed)
mentioned in merge request !103484 (merged)
mentioned in issue gitaly#4615 (closed)
added releasedcandidate label
added releasedpublished label and removed releasedcandidate label
mentioned in merge request !107682 (merged)
mentioned in merge request gitaly!5937 (merged)
mentioned in issue #438553 (closed)