Pin the minor version of Security Products (tools and analyzers) in the vendored templates

Problem to solve

Updating every instances in the world at once each time we release a new version of our Security Products is risky.

While this gives us more flexibility to push updates it also could cause huge troubles because with time it's really hard to ensure every pieces will keep working seamlessly together. We also don't have a fully backward compatibility testing process, e.g. we don't test our new released tools with GitLab 10.x or 11.x.

This is also particularly true as we currently don't have a process to publish release candidates.

Further details

For main tools (SAST, Dependency Scanning, Container Scanning, License Management) we override the Major docker image tag every month when there is a new version of GitLab. All these Major images are actually acting as a compatibility matrix layer.

When we release a new version of a tool:

1. we publish a new semantic version with a git tag: 1.2.4
2. we publish a corresponding docker image: sast:1.2.4
3. we override the corresponding major docker image: sast:1

The step 3 is the main issue and the behavior we need to change.

Proposal

To address the issue of having all instances in the world pulling instantly the latest version of our tool, we should pin to the minor version of our analyzers instead of the major.

Here is an example of how it would end up:

• GitLab 11.8 uses the sast:1.3 docker image.
• We publish sast 1.4.0, which also creates a sast:1.4 docker image.
• GitLab 11.8 still uses sast 1.3 has the sast:1.3 docker image has not changed.
• We update the vendored include to use sast:1.4 and when GitLab 11.9 is released, it uses sast 1.4.

If we have a bug to fix:

• We publish sast 1.3.1, which also override the sast:1.3 docker image.
• GitLab 11.8 immediately get the patched version

Pros and Cons

Pros:

1. Make it simpler for all stakeholders:
   • release process and CI configuration will be simpler and more straightforward
   • each version of GitLab has a specific version of the Security Tools explicitly listed in the vendored template
   • newcomers won't have to deal with this unusual versioning and spend time figuring out how it works
2. this is a common behavior when dealing with dependencies: pinning a minor to get all patches automatically but not the new features to avoid unwanted changes. It makes sense to ask our users to update to the latest version of GitLab to benefit from new features and it reduces the risk of breaking things as time goes. At the pace of changes happening on the product, this seems to be a safer approach.
3. allow us moving forward quicker than when trying to keep all changes fully compatible with old releases of GitLab. Remember, we publish a major release only once a year now.

Cons:

1. by pinning to major 2 docker images, we're pushing the latest update of our tools, including the vulnerability rules/definitions to all customers. This means there is currently no need to update GitLab to the latest monthly release to benefit from up-to-date vulnerability detection capabilities. If we break this, what's the plan to keep the old instances up-to-date?
   • Olivier: let's investigate on this. How many tools/analyzers are concerned with this (some of them already benefit from live update at runtime). We could also look for how many times these concerned tools/analyzers have been updated since they've been added. Ultimately, this kind of update could end up in the backport process. E.g. pushing the update to 1.5.x, 1.4.x, 1.3.x, etc.
2. pushing updates to our tools is becoming more dependent on the Rails application release process as we need to bump the pinned version in the vendored template. Though, this also come with more stability and robustness as we ensure full QA before using a new release. And as mentioned in the proposal, by pining to a Minor, we can still push patches immediately without going through a release of the GitLab Rails application.
3. as we officially need to support the last 3 major versions of Gitlab, this can imply some overhead in maintenance and we probably should have a well-defined guideline about how to release our tool: When should I bump a minor or a patch? What needs to be backported to previous minor releases? How far? And how to do it?

Tasks:

• update the vendored templates https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security to use the new major.minor image tag instead of major
  • SAST.gitlab-ci.yml
  • Dependency-Scanning.gitlab-ci.yml
  • License-Scanning.gitlab-ci.yml
  • Container-Scanning.gitlab-ci.yml
  • Secure-Binaries.gitlab-ci.yml
• update the release process documentation to add a step to update the vendored template when releasing a tool or an analyzer
• update the CI config for SAST, DS, CS, and LM
  • build the major.minor docker image tag along with major.minor.patch and major ones.
  • stop releasing x-y-stable Docker tags for tag builds

What does success look like, and how can we measure that?

We publish docker images matching semantic versions of our tools

What is the type of buyer?

GitLab Ultimate

Links / references

Product documentation to be updated