Backend: Organize and define behavior of jobs templates to be included
Problem to solve
With the implementation of Composable Auto DevOps some job definitions have been placed in a Jobs/
subfolder: https://gitlab.com/gitlab-org/gitlab-ee/tree/master/lib/gitlab/ci/templates/Jobs
This has caused few troubles which demonstrate the need to enforce some rules (with tests) and provide guidelines for other teams to leverage the include feature.
Intended users
Developers
Further details
Here are assumptions from the experience of the ~Secure team when Vendoring the template for Security Products that need to be confirmed or adjusted:
-
Template name must be unique, whatever the subfolder it is placed in. E.g. having these two templates should result in a test failure:
Security/DAST.gitlab-ci.yml
Jobs/DAST.gitlab-ci.yml
This rule was motivated by the behavior explained in
#2
. -
Template placed in a subfolder can be included without specifying the folder. E.G
include: - template: SAST.gitlab-ci.yml
The gotcha we just discovered is that for #2
to work, the folder actually has to be listed as a category. For Security
templates, this was done in EE only by extending the list. Though having the folder listed as a category also has the side effect to display the templates it contains in the Template dropdown in GitLab UI. For this reason (AFAIR), the Jobs
template where not listed as a category.
The resulting behavior is:
- in CE,
Security
templates must be included using the folder prefix:template: Security/SAST.gitlab-ci.yml
(though, we don't particularly want to include them in CE, all our features are premium tiers) - in EE,
Security
templates can be included both with or without the folder prefix:template: Security/SAST.gitlab-ci.yml
ortemplate: SAST.gitlab-ci.yml
- in CE and EE,
Jobs
templates must be included using the folder prefix:template: Jobs/Build.gitlab-ci.yml
NB: The ~Secure feature have been vendored in 11.9
following these assumptions listed above and we need to keep them working as is to avoid breaking customer configs. At least, if we want to change the behavior, we need to deprecate it now and drop
support with 12.0
.
Proposal
As an urgent fix (before April 7th)
The Jobs/DAST.gitlab-ci.yml
must be removed to only keep the existing Security/SAST.gitlab-ci.yml
.
This is addressed by this MR: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26957
As a short term solution (before April 22nd?/release of 11.10
)
We need to move Security templates into Jobs subfolder to benefit from further improvements regarding job templates
We need to decide which behavior we want to provide and align all our job templates on it.
Do we want to keep allowing inclusion of templates without specifying the subfolder?
- YES:
- rule
#1
must be enforced, by adding a unit test ensuring uniqueness of template names, whatever their subfolder (both in CE and EE), and document this.
- rule
- NO:
- we need to announce deprecation of what ~Secure has released in
11.9
, which is using unprefixed include e.g.template: SAST.gitlab-ci.yml
and promote prefixed usage everywhere. Though, to avoid redoing a change later, I'd highly recommend that instead of asking to use theSecurity/
prefix we actually move Security templates into Jobs subfolder directly. - we need to decouple the logic that allows unprefixed include from the one that exposes the templates in the UI (dropdown), so that without exposing the
Jobs/
templates in the UI, we still allow to include them unprefixed (to avoid breaking change for what ~Secure has released in11.9
). This code could then be dropped in12.0
to disallow unprefixed include, if not used anywhere else.
- we need to announce deprecation of what ~Secure has released in
What does success look like, and how can we measure that?
Unified behavior for job templates, at least from 12.0
if it implies breaking changes for what has been already released.