Sometimes you want to define rules to decide whether an entire pipeline is run. The only way to do that now is to add the duplicate criteria to every single job in the pipeline, and then if none of them end up resolving to true, the entire pipeline is skipped. It would be more efficient if there was a global setting for a pipeline that could be evaluated first.
This would also help with composability where you have less influence on the per-job rules, which could potentially cause issues where those jobs run even when you don't want anything to happen.
Another good example of where this is difficult today is with the only: merge_requests rule. You may always want to run every job for a merge request too, but there is no easy way to do this except by copy-pasting the only: merge_requests value into every single job.
Intended users
Software Developers
Proposal
We will add a new top level workflow: section that supports a single entity inside, a rules: section. In order to determine if a pipeline should be triggered at all, this is evaluated first:
workflow:rules:# this rule says that the entire pipeline should only run for merge requests, or for `master`.-branch:master-merge-request:yesonly_changes:script:...rules:# this job will only run on changes to .rb files because it has its own rule stating so:-changes:*.rbalways_run:# this job will always run as long as the pipeline is running, so doesn't need to add any rules.script:...
Note that the top level workflow: evaluation is independent of the job rules: or only/except definitions. It is not re-evaluated on a per-job basis. If a job has a rules: section, it is evaluated independently of the workflow:rules section at the time the job is started.
Also, this feature builds on top of our new rules: syntax implemented here: https://gitlab.com/gitlab-org/gitlab-ce/issues/60085, but only implements the when:never: rules (if no when: is added, it treats it as a match and moves on.)
Permissions and Security
Documentation
In addition to documenting this feature on its own, we also need to update the pipelines for merge requests documentation to reference that this example can be useful for avoiding two pipelines being created (i.e., when there's a mix of merge request and general pipeline jobs):
workflow:rules:-branch:master-merge-request:yes
Testing
What does success look like, and how can we measure that?
Jason Yavorskachanged title from Provide for global rule-setting in .gitlab-ci.yml to Provide for global {+rules: +}setting in .gitlab-ci.yml
changed title from Provide for global rule-setting in .gitlab-ci.yml to Provide for global {+rules: +}setting in .gitlab-ci.yml
Jason Yavorskachanged title from Provide for global {-rules:-} setting in .gitlab-ci.yml to Provide for global only/except or rules setting in .gitlab-ci.yml
changed title from Provide for global {-rules:-} setting in .gitlab-ci.yml to Provide for global only/except or rules setting in .gitlab-ci.yml
@ayufan@drewcimino I've updated this description to re-add the workflow: top-level element. A couple questions:
I've left the door open here also for defining only/except rules in this same place, that are also inherited in the same way. Is there any problem with this?
Adding a global rules: or only/except section will cause all jobs in the pipeline to inherit that ruleset. At this point we've "tainted" every job, meaning that all includes and jobs must now consistently use one or the other. We explicitly implemented flexible rules on a per-job basis to avoid this situation. Is there a clever way to avoid it here?
I've left the door open here also for defining only/except rules in this same place, that are also inherited in the same way. Is there any problem with this?
Not sure if I understand. I would say we never allow to define global only/except there's no point of doing so.
Adding a global rules: or only/except section will cause all jobs in the pipeline to inherit that ruleset. At this point we've "tainted" every job, meaning that all includes and jobs must now consistently use one or the other. We explicitly implemented flexible rules on a per-job basis to avoid this situation. Is there a clever way to avoid it here?
No, no. The idea behind workflow:rules: is that this is evaluated before, for the whole Pipeline, not for the per-individual job. It means that setting is not inherited, rather job has it's own specification that is run in the context of the job. It serves, a case that you want this .gitlab-ci.yml to run in very specific cases only, and this applies to all, as you have a very specific workflow, like git flow, everything else should be rejected as far as CI goes. It means, that this is evaluated first. It also means that the job could use rules or only/except as this is then executed in the context of the job.
This also means that we would continue supporting all today's jobs.
This will continue working after introducing workflow:rules:.
There was the concept of adding default:rules:. Conceptually default: is different,
it allows you to define a default configuration for job, that can be rewritten (not extended)
by job, so it serves a little like an implicit extends: default. It is not very well suited
to add a composable list of rules, where we join two specifications together. I would consider
extending default: with as many of other configurations as possible that are part now of job
definition, but cannot be defined globally, like tags: (ability to select default runner for all
CI jobs).
@ayufan can you please re-read the issue description and make sure that it makes sense? Can you also specifically expand on how the only: merge_requests usage is improved, since that was a main driver of why we are prioritizing this feature?
pipeline_when=pipeline.workflow_rules.evalaute(variables,default_when: :on_success)# when: on_success|never|...ifpipeline_when==:never# do not create pipeline at allreturnendpipelines.jobs.eachdo|job|run_when=job.rules.evaluate(variables,default_when: pipeline_when||:on_success)ifrun_when!=:neverjob.create!(when: run_when)endend
The logic is that we inherit the workflow:rules:when if the when: is not specified for a given job-level rule. If it is specified the job level rule when takes precedence.
The logic to describe the above pseudo-code:
workflow:rules:-branch:/^stable-/when:never-branch:master# it defines implicit `when: on_success`-tag:yeswhen:alwaysrspec:rules:-branch:masterchanges:*.rb-tag:/v1/when:on-success-tag:/v2/-tag:yeswhen:never
push to stable-10: branch: /^stable- => when: never => do not create pipeline
push to master: branch: master => when: on_succes (from workflow:rules) => rspec:rules matching changes: *.rb => inherits workflow:rules:when (in this case on_success)
push to v1: tag: yes => when: always (from workflow:rules) => rspec:rules matching tag: /v1/, it overwrites when to on-success => uses when: on-success
push to v2: tag: yes => when: always (from workflow:rules) => rspec:rules matching tag: /v2/, it does not have when defined in rspec:rules, so it inherits when: always => uses when: always
push to v3: tag: yes => when: always (from workflow:rules) => rspec:rules matching tag: yes, it does have when: never => do not create job
Jason Yavorskachanged title from Provide for global only/except or rules setting in .gitlab-ci.yml to Global only/except/rules setting in .gitlab-ci.yml
changed title from Provide for global only/except or rules setting in .gitlab-ci.yml to Global only/except/rules setting in .gitlab-ci.yml
Hi folks! Quick question, is there a reason we put this under the workflow keyword? Are there other flow control features we would want to add here? It is a bit different than the job definition, where you don't need it.
@joshlambert it's to make it clear that the rules: section is applicable to the entire workflow, as opposed to being something re-evaluated in every job. This is an addition (special case) for the rules: syntax being implemented in https://gitlab.com/gitlab-org/gitlab-ce/issues/60085.
What does the merge-requests keyword mean exactly? Is it an addition or a restriction? I want a pipeline created for any branch push and for merge requests, but only one (except if a MR is created after a branch has been pushed, of course). And I want branch pipelines to contain job A, and MR pipelines to contain jobs A and B. And pushes to master (i.e., upon merging a MR) to contain job C. Can such a thing be obtained with this proposal?
only: ['master'] or only: ['branches'] works as expected. Not sure why merge_request is the outlier here and spawns a new pipeline outside of the stages flow.
It is outliner, because of backward compatibility. We try very hard to not change existing behavior. This is why we introduce a new more flexible construct.
Let's consider that we want to ensure that the following workflow will work for merge-request-pipelines (and merge trains):
workflow: rules: - merge_requests: yesmy_job: script: echo Hello World
Today, if we introduce workflow:rules: it will not work properly, as my_job: has implicit only: definition to be [branches,tags]. This makes the workflow:rules: to evaluate true for Merge Request pipelines, but fail to validate my_job.
This means that if we have global workflow:rules: we should completely ignore my_job:only/except the implicit values unless it is explicitly defined.
GitLab is moving all development for both GitLab Community Edition
and Enterprise Edition into a single codebase. The current
gitlab-ce repository will become a read-only mirror, without any
proprietary code. All development is moved to the current
gitlab-ee repository, which we will rename to just gitlab in the
coming weeks. As part of this migration, issues will be moved to the
current gitlab-ee project.
If you have any questions about all of this, please ask them in our
dedicated FAQ issue.
Using "gitlab" and "gitlab-ce" would be confusing, so we decided to
rename gitlab-ce to gitlab-foss to make the purpose of this FOSS
repository more clear
I created a merge requests for CE, and this got closed. What do I
need to do?
Everything in the ee/ directory is proprietary. Everything else is
free and open source software. If your merge request does not change
anything in the ee/ directory, the process of contributing changes
is the same as when using the gitlab-ce repository.
Will you accept merge requests on the gitlab-ce/gitlab-foss project
after it has been renamed?
No. Merge requests submitted to this project will be closed automatically.
Will I still be able to view old issues and merge requests in
gitlab-ce/gitlab-foss?
Yes.
How will this affect users of GitLab CE using Omnibus?
No changes will be necessary, as the packages built remain the same.
How will this affect users of GitLab CE that build from source?
Once the project has been renamed, you will need to change your Git
remotes to use this new URL. GitLab will take care of redirecting Git
operations so there is no hard deadline, but we recommend doing this
as soon as the projects have been renamed.
Where can I see a timeline of the remaining steps?