Additional requirements to the include feature to make Security Products projects able to use it
Description
The include
keyword should be used by ~"Security Products" projects to avoid repetitions and to allow easy updates. At the moment, there are a few blockers that don't allow that, and we should improve the feature where possible.
Since we are our first customer, this issue will list and discuss all the possible requests/constraints in order to evaluate the feasibility and to create a plan.
Problems
1. Versioned includes
Description
To ensure compatibility of the template with the current version running on the instance, we must provide versioned includes.
Workarounds
- just include the
master
template for now and restrict it usage to test projects and things we don't expose to our users. - use hardcoded version and update on each release (yes, we'll forget :D)
Solution
A - Variable expansion
To achieve this we need to have the value(s) of the include
property supporting variable expansion so that we could do something like:
include: https://gitlab.com/gitlab-org/gitlab-ci-yml/raw/$GITLAB_STABLE_VERSION/includes/security-products/sast.gitlab-ci.yml
Having GITLAB_STABLE_VERSION
filled with 10-7-stable
to fetch the template on the corresponding release branch.
B - Expose GITLAB_STABLE_VERSION
Using a variable within an include
also means we need to expose that value in a specific way. E.g. it cannot be done in a before_script
as the include
are evaluated before. Having this available directly as Predefined variable should solve the issue. Still, we need to clearly define the naming convention. E.g. with Gitlab 10.7.1-ee
we could have GITLAB_STABLE_VERSION
filled with:
10-7-stable
10.7
## 2. Job detection to show MR widget reports
### Description
The MR widget shows reports for features that are enabled and this detection is based on the job name and the presence of matching artifacts. Though it looks like the detection only works when the job is declared in te main template. When declared in an include, the MR widget doesn't show the report, even if the job ran successfully and artifacts are available.
### Solution
Update the detection logic in Gitlab backend code.
3. Job customization
Description
Some jobs need to be tuned on some projects and as yaml anchors won't be available through includes this means the whole job as to be overridden.
Workaround
- override the whole job (which makes the include totally worthless)
Solution
Using a deep merge instead of shallow merge when using includes is currently being implemented: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5288