Pin the minor version of Security Products (tools and analyzers) in the vendored templates
Problem to solve
Updating every instances in the world at once each time we release a new version of our Security Products is risky.
While this gives us more flexibility to push updates it also could cause huge troubles because with time it's really hard to ensure every pieces will keep working seamlessly together. We also don't have a fully backward compatibility testing process, e.g. we don't test our new released tools with GitLab 10.x
or 11.x
.
This is also particularly true as we currently don't have a process to publish release candidates.
Further details
For main tools (SAST, Dependency Scanning, Container Scanning, License Management) we override the Major
docker image tag every month when there is a new version of GitLab.
All these Major
images are actually acting as a compatibility matrix layer.
When we release a new version of a tool:
- we publish a new semantic version with a git tag:
1.2.4
- we publish a corresponding docker image:
sast:1.2.4
- we override the corresponding major docker image:
sast:1
The step 3
is the main issue and the behavior we need to change.
Proposal
To address the issue of having all instances in the world pulling instantly the latest version of our tool, we should pin to the minor
version of our analyzers instead of the major
.
Here is an example of how it would end up:
- GitLab
11.8
uses thesast:1.3
docker image. - We publish sast
1.4.0
, which also creates asast:1.4
docker image. - GitLab
11.8
still uses sast1.3
has thesast:1.3
docker image has not changed. - We update the vendored include to use
sast:1.4
and when GitLab11.9
is released, it uses sast1.4
.
If we have a bug to fix:
- We publish sast
1.3.1
, which also override thesast:1.3
docker image. - GitLab
11.8
immediately get the patched version
Pros and Cons
Pros:
- Make it simpler for all stakeholders:
- release process and CI configuration will be simpler and more straightforward
- each version of GitLab has a specific version of the Security Tools explicitly listed in the vendored template
- newcomers won't have to deal with this unusual versioning and spend time figuring out how it works
- this is a common behavior when dealing with dependencies: pinning a minor to get all patches automatically but not the new features to avoid unwanted changes. It makes sense to ask our users to update to the latest version of GitLab to benefit from new features and it reduces the risk of breaking things as time goes. At the pace of changes happening on the product, this seems to be a safer approach.
- allow us moving forward quicker than when trying to keep all changes fully compatible with old releases of GitLab. Remember, we publish a major release only once a year now.
Cons:
- by pinning to major
2
docker images, we're pushing the latest update of our tools, including the vulnerability rules/definitions to all customers. This means there is currently no need to update GitLab to the latest monthly release to benefit from up-to-date vulnerability detection capabilities. If we break this, what's the plan to keep the old instances up-to-date?- Olivier: let's investigate on this. How many tools/analyzers are concerned with this (some of them already benefit from live update at runtime). We could also look for how many times these concerned tools/analyzers have been updated since they've been added. Ultimately, this kind of update could end up in the backport process. E.g. pushing the update to
1.5.x
,1.4.x
,1.3.x
, etc.
- Olivier: let's investigate on this. How many tools/analyzers are concerned with this (some of them already benefit from live update at runtime). We could also look for how many times these concerned tools/analyzers have been updated since they've been added. Ultimately, this kind of update could end up in the backport process. E.g. pushing the update to
- pushing updates to our tools is becoming more dependent on the Rails application release process as we need to bump the pinned version in the vendored template. Though, this also come with more stability and robustness as we ensure full QA before using a new release. And as mentioned in the proposal, by pining to a Minor, we can still push patches immediately without going through a release of the GitLab Rails application.
- as we officially need to support the last 3 major versions of Gitlab, this can imply some overhead in maintenance and we probably should have a well-defined guideline about how to release our tool: When should I bump a
minor
or apatch
? What needs to be backported to previous minor releases? How far? And how to do it?
Tasks:
-
update the vendored templates https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security to use the new major.minor
image tag instead ofmajor
-
SAST.gitlab-ci.yml
-
Dependency-Scanning.gitlab-ci.yml
-
License-Scanning.gitlab-ci.yml
-
Container-Scanning.gitlab-ci.yml
-
Secure-Binaries.gitlab-ci.yml
-
-
update the release process documentation to add a step to update the vendored template when releasing a tool or an analyzer -
update the CI config for SAST, DS, CS, and LM -
build the major.minor
docker image tag along withmajor.minor.patch
andmajor
ones. -
stop releasing x-y-stable
Docker tags for tag builds
-
What does success look like, and how can we measure that?
We publish docker images matching semantic versions of our tools