Reconsider the versioning of SAST

Problem to solve

This issue is the counterpart https://gitlab.com/gitlab-org/gitlab-ee/issues/6797 for the orchestrator.

The current release process for Security Products implies to generate a new docker image for each major or minor release of GitLab. For that, each major-minor version is managed in a dedicated branch. While old versions of GitLab are not supported after a while, the Security checks are expected to still work, and for that, they need to be up-to-date. Maintaining many old versions will get harder and harder, to a point where even the slightest change will require too much effort.

Further details

To allow changes in the output format, we have scoped every sast version to a major-minor version of GitLab. This allowed us to introduce breaking changes in the format, and the downside is the requirement to maintain many branches and docker images. Cherry-picking changes can be tedious and error-prone, especially with many branches. Last but not least, we're making our QA process more complex than it has to be. The only interaction between GitLab and SAST is the CI job running, and the job JSON output (parsed and used by the front end).

Proposal

Docker tags for GitLab versions

When building Docker images of SAST, the image tag corresponds to the version of GitLab we're targeting. It helps to automatically select (ie. download and run) the right version of SAST.

This is already the way it works so there's no need to change the job definition of SAST. It's backward compatible with existing job definitions.

See current definition of the sast job:

sast:
  image: docker:stable
  variables:
    DOCKER_DRIVER: overlay2
  allow_failure: true
  services:
    - docker:stable-dind
  script:
    - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')
    - docker run
        --env SAST_CONFIDENCE_LEVEL="${SAST_CONFIDENCE_LEVEL:-3}"
        --volume "$PWD:/code"
        --volume /var/run/docker.sock:/var/run/docker.sock
        "registry.gitlab.com/gitlab-org/security-products/sast:$SP_VERSION" /app/bin/run /code
  artifacts:
    paths: [gl-sast-report.json]

Semantic versioning

Under the hood, SAST would have its own versioning, based on format breaking changes. For example, the version 1 would be updated as long as the format is compatible with the previous version. More features, stability, bug fixes, ... are getting added to this version.

When we break the compatibility of the output:

  • We create a new version X+1
  • we create a version X branch
  • we upgrade the ci configuration (see below).

Building the Docker images

For each MAJOR version of SAST, the CI builds and tags multiple Docker images targeting various versions of GitLab.

That way, we ensure all changes are getting published with no effort to previous, compatible versions, for example:

graph TD;
  V1-->10.4;
  V1-->10.5;
  V1-->10.6;
  V1-->10.7;
  V2-->10.8;
  V2-->11.0;
  V2-->11.1;
  V2-->11.2;

Enabling the analyzers

As mentioned above, multiple Docker images targeting multiple versions of GitLab (like 10.8, 11.0, 11.1) will share the same codebase (like SAST v1). This is OK but we've got to make sure that we don't backport some analyzers introduced recently to old versions of GitLab (where they were not present initially). In other words, the behavior of SAST has to be consistent with the changelog. To achieve that, we introduce environment variables and Docker build args to explicitly enable or disable analyzers when building the Docker images.

Fetching the analyzers

Reminder: The common lib establishes how SAST interacts with its analyzers (cli arguments, output format, etc.). As a consequence, the version of common is in fact the version of the protocol/contract, and SAST will be able to interact with an analyzer as long as they both share the same MAJOR version.

SAST will fetch the latest compatible versions of its analyzers. To achieve this, each analyzer must be published as a Docker image whose tag correspond to the MAJOR number. SAST v2 will fetch bandit:2, flawfinder:2, etc.

How to migrate

The latest version should be managed under the master branch. All previous version are managed under their respective v[x] branch.

The key to achieving this is the CI configuration.

In the following example, the master branch corresponds to SAST v2. When pushing to master, the CI builds the Docker images sast:10-8-stable to sast:11-2-stable. NodeJs-Scan is enabled starting from sast:11-1-stable, thus only available in GitLab 11.1 and later. When pushing to the v1 branch, the CI builds sast:10-7-stable and older.

.build_base: &build_base
  [...]
  variables:
    - SAST_ANALYZER_NODEJS_SCAN=0
    [...]
  script:
     [...]
     - JOB_NAME=( $CI_JOB_NAME )
     - docker build --build-arg SAST_ANALYZER_NODEJS_SCAN=$SAST_ANALYZER_NODEJS_SCAN [...]
     - docker tag sast:$JOB_NAME
     - docker push sast:$JOB_NAME
     [...]
  only:
    - master

build 10-8-stable: *build_base
build 11-0-stable: *build_base

build 11-1-stable: *build_base
  variables:
    - SAST_ANALYZER_NODEJS_SCAN=1

build 11-2-stable: *build_base
  variables:
    - SAST_ANALYZER_NODEJS_SCAN=1

.build_v1: &build_v1_base
  [...]
  script:
     [...]
     - JOB_NAME=( $CI_JOB_NAME )
     - docker build [...]
     - docker tag sast:$JOB_NAME
     - docker push sast:$JOB_NAME
     [...]
  only:
    - v1

build 10-4-stable: *build_v1_base
build 10-5-stable: *build_v1_base
build 10-6-stable: *build_v1_base
build 10-7-stable: *build_v1_base

We can also instantly understand what version of SAST serves which version(s) of GitLab.

Update release process

The release process will have to be adapted for this change:

  • On the 8th, the next version must be added to the list of build jobs (in gitlab-ci.yml)
  • The QA process will just test the latest version (older versions are tested upon merge)
  • No new branch is created

What does success look like, and how can we measure that?

  • Pushing a new commit to master creates all related backwards compatible docker images. All these images share the same image ID.
  • Pushing a new commit to old branches creates all related backwards compatible docker images. All these images share the same image ID.
  • QA is only related to a version of SAST, not especially tight to a gitlab version.
Edited by Fabien Catteau