Define a policy for Secure analyzer images update cadence
Problem to solve
The AppSec team has asked our whole Stage to adopt a nightly build/release cycle for our security analyzer container images.
Between all the devopssecure groups, there are different approaches to how and when container images are updated.
As part of fulfilling the request from AppSec, we would like to take the opportunity to align the update policy across our stage.
Related OKR:
https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5377- https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5901
Proposal
What
The requirement is to rebuild the analyzer's docker images on daily basis to pull upstream updates of the dependencies bundled within the base image. The process should be automated and require minimal to no maintenance overhead to the engineering team. If this is not doable, exception might be requested on a per project basis by the team and reviewed with Appsec.
Unless it's necessary, this requirement only applies to the images used in versions of GitLab matching the current MAJOR release. Though, to stay compliant with our release and maintenance policy, we will need to continue rebuilding the previous major version of the analyzer right after the transition to a new major version of GitLab. This will have be observed during 2 months (X.0 and X.1).
Why
The main benefit is to increase our maintenance and security posture by frequently and automatically pulling patches provided by vendors of the base images our analyzers rely upon.
The development teams are responsible for addressing the detected vulnerabilities within SLOs (SLAs for FedRAMP). If we don't have automatic rebuild, we have more manual work to fulfill this requirement. Automation are also being added company wide to execute scans and create security issues which might soon overwhelm development teams if such work stays manual.
So, as a by product, automatic rebuild will reduce the cost of managing the security vulnerabilities detected by the Container Scanning analyzer on these images. Indeed, as we continuously update the base image, some findings will automatically be fixed without action from the development team.
How
Building the images
Each variant of an image must provide the following tags:
-
MAJOR.MINOR.PATCH
(e.g.:4.1.7
) -
MAJOR.MINOR
(e.g.:4.1
) -
MAJOR
(e.g.:4
) latest
- TBC:
MAJOR.MINOR.PATCH-DATE
: (e.g.:4.1.7-20240126
)
The tag MAJOR.MINOR.PATCH-DATE
: (e.g.: 4.1.7-20240126
) is yet to be confirmed. The intent is to provide persisting references to the daily builds. While this could be helfpul to debug some problems, this drastically increases the number of image tags we need to keep in the registry. See ongoing discusssion in this thread: #429152 (comment 1748087540)
Daily rebuild
Each variant of an image must be rebuilt on a daily basis and:
- override the corresponding
MAJOR.MINOR.PATCH
(e.g.:4.1.7
) - override the corresponding
MAJOR.MINOR
(e.g.:4.1
) - override the corresponding
MAJOR
(e.g.:4
) - override
latest
- TBC: build a new
MAJOR.MINOR.PATCH-DATE
tag: (e.g.:4.1.7-20240127
)
Implementation
The shared CI configuration in https://gitlab.com/gitlab-org/security-products/ci-templates/-/blob/master/includes-dev/docker.yml?ref_type=heads currenlty allows to easily build and rebuild a docker image with the expected matching tag requirements (except MAJOR.MINOR.PATCH-DATE
). This CI configuraton relies on some prerequisites (like having a CHANGELOG.MD file using semver compliant versions) and might require advanced tunning depending on the project needs. Scheduled pipelines should be configured to do the daily rebuild.
Note 1: There is an improvement to be made on this existing process to prevent race conditions problems with the temporary image build in the pipeline. Adding a reference like CI_PIPELINE_ID
in the tagging scheme should address this (see #429152 (comment 1695685506)). This is not a blocker for moving forward.
Note 2: There is a potential race condition problem that could cause the tag latest
to not point to the last major release when a project keeps building multiple major versions. This is not a blocker for moving forward.
Follow-ups
This policy focuses on the simplest path forward and the scope has been limited on purpose. We should consider defining a policy for dependencies not covered by this nightly rebuild process. Some relevant details in #429152 (comment 1643424006)
Intended users
Feature Usage Metrics
Does this feature require an audit event?
/cc @amarpatel @twoodham