Reconsider the versioning of SAST
Problem to solve
This issue is the counterpart https://gitlab.com/gitlab-org/gitlab-ee/issues/6797 for the orchestrator.
The current release process for Security Products implies to generate a new docker image for each major or minor release of GitLab. For that, each major-minor version is managed in a dedicated branch. While old versions of GitLab are not supported after a while, the Security checks are expected to still work, and for that, they need to be up-to-date. Maintaining many old versions will get harder and harder, to a point where even the slightest change will require too much effort.
Further details
To allow changes in the output format, we have scoped every sast version to a major-minor version of GitLab. This allowed us to introduce breaking changes in the format, and the downside is the requirement to maintain many branches and docker images. Cherry-picking changes can be tedious and error-prone, especially with many branches. Last but not least, we're making our QA process more complex than it has to be. The only interaction between GitLab and SAST is the CI job running, and the job JSON output (parsed and used by the front end).
Proposal
Docker tags for GitLab versions
When building Docker images of SAST, the image tag corresponds to the version of GitLab we're targeting. It helps to automatically select (ie. download and run) the right version of SAST.
This is already the way it works so there's no need to change the job definition of SAST. It's backward compatible with existing job definitions.
See current definition of the sast
job:
sast:
image: docker:stable
variables:
DOCKER_DRIVER: overlay2
allow_failure: true
services:
- docker:stable-dind
script:
- export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')
- docker run
--env SAST_CONFIDENCE_LEVEL="${SAST_CONFIDENCE_LEVEL:-3}"
--volume "$PWD:/code"
--volume /var/run/docker.sock:/var/run/docker.sock
"registry.gitlab.com/gitlab-org/security-products/sast:$SP_VERSION" /app/bin/run /code
artifacts:
paths: [gl-sast-report.json]
Semantic versioning
Under the hood, SAST would have its own versioning, based on format breaking changes. For example, the version 1 would be updated as long as the format is compatible with the previous version. More features, stability, bug fixes, ... are getting added to this version.
When we break the compatibility of the output:
- We create a new version X+1
- we create a version X branch
- we upgrade the ci configuration (see below).
Building the Docker images
For each MAJOR version of SAST, the CI builds and tags multiple Docker images targeting various versions of GitLab.
That way, we ensure all changes are getting published with no effort to previous, compatible versions, for example:
graph TD;
V1-->10.4;
V1-->10.5;
V1-->10.6;
V1-->10.7;
V2-->10.8;
V2-->11.0;
V2-->11.1;
V2-->11.2;
Enabling the analyzers
As mentioned above, multiple Docker images targeting multiple versions of GitLab (like 10.8, 11.0, 11.1) will share the same codebase (like SAST v1). This is OK but we've got to make sure that we don't backport some analyzers introduced recently to old versions of GitLab (where they were not present initially). In other words, the behavior of SAST has to be consistent with the changelog. To achieve that, we introduce environment variables and Docker build args to explicitly enable or disable analyzers when building the Docker images.
Fetching the analyzers
Reminder: The common
lib establishes how SAST interacts with its analyzers (cli arguments, output format, etc.).
As a consequence, the version of common
is in fact the version of the protocol/contract, and SAST will be able to interact with an analyzer as long as they both share the same MAJOR version.
SAST will fetch the latest compatible versions of its analyzers. To achieve this, each analyzer must be published as a Docker image whose tag correspond to the MAJOR number. SAST v2 will fetch bandit:2
, flawfinder:2
, etc.
How to migrate
The latest version should be managed under the master
branch.
All previous version are managed under their respective v[x]
branch.
The key to achieving this is the CI configuration.
In the following example, the master
branch corresponds to SAST v2. When pushing to master
, the CI builds the Docker images sast:10-8-stable
to sast:11-2-stable
. NodeJs-Scan is enabled starting from sast:11-1-stable
, thus only available in GitLab 11.1 and later. When pushing to the v1
branch, the CI builds sast:10-7-stable
and older.
.build_base: &build_base
[...]
variables:
- SAST_ANALYZER_NODEJS_SCAN=0
[...]
script:
[...]
- JOB_NAME=( $CI_JOB_NAME )
- docker build --build-arg SAST_ANALYZER_NODEJS_SCAN=$SAST_ANALYZER_NODEJS_SCAN [...]
- docker tag sast:$JOB_NAME
- docker push sast:$JOB_NAME
[...]
only:
- master
build 10-8-stable: *build_base
build 11-0-stable: *build_base
build 11-1-stable: *build_base
variables:
- SAST_ANALYZER_NODEJS_SCAN=1
build 11-2-stable: *build_base
variables:
- SAST_ANALYZER_NODEJS_SCAN=1
.build_v1: &build_v1_base
[...]
script:
[...]
- JOB_NAME=( $CI_JOB_NAME )
- docker build [...]
- docker tag sast:$JOB_NAME
- docker push sast:$JOB_NAME
[...]
only:
- v1
build 10-4-stable: *build_v1_base
build 10-5-stable: *build_v1_base
build 10-6-stable: *build_v1_base
build 10-7-stable: *build_v1_base
We can also instantly understand what version of SAST serves which version(s) of GitLab.
Update release process
The release process will have to be adapted for this change:
- On the 8th, the next version must be added to the list of build jobs (in
gitlab-ci.yml
) - The QA process will just test the latest version (older versions are tested upon merge)
- No new branch is created
What does success look like, and how can we measure that?
- Pushing a new commit to master creates all related backwards compatible docker images. All these images share the same image ID.
- Pushing a new commit to old branches creates all related backwards compatible docker images. All these images share the same image ID.
- QA is only related to a version of SAST, not especially tight to a gitlab version.