Move build of gemnasium-python, gemnasium-maven to gemnasium project
Summary
To reduce maintenance cost, move the build of gemnasium-maven and gemnasium-python to gemnasium, so that this project builds all Docker images related to Gemnasium: gemnasium
, gemnasium-maven
, and gemnasium-python
.
As a result, gemnasium-maven and gemnasium-python are no longer needed, and they are archived.
This is a follow-up to #288063 (closed) (sharing the codebase), and is blocked by it.
Proposal
Doing the following as we release Gemnasium v3:
- Update the
gemnasium
project to build, test, and release all 3 Docker images corresponding togemnasium
,gemnasium/maven
, andgemnasium/python
. This is implemented using child pipelines. - Update the README of
gemnasium
to reflect this change, and reset the CHANGELOG. - Keep gemnasium-maven and gemnasium-python to maintain v2 branch, but update the README to indicate that v3 lives in
gemnasium
.
gemnasium-maven:3
and gemnasium-python:3
are published in the SEC_REGISTRY
, but not in the Container Registry of gemnasium-maven
and gemnasium-python
.
The temporary images are gemnasium/tmp
, gemnasium/maven/tmp
, and gemnasium/python/tmp
. These images are pushed to the Container Registry of the gemnasium
project.
The gemnasium
project has a parent pipeline that runs the unit tests and lint the code, and child pipelines that build, test, and release the images, using trigger
jobs. It includes go.yml
and upsert-git-tag.yml
.
Each child pipeline corresponds to a "build directory" that contains Dockerfile
, main.go
, .gitlab-ci.yml
, and files that are specific to the build. For instance, build/gemnasium-python
contains the files needed to build the gemnasium-python
image.
Child pipelines include docker.yml
and docker-test.yml
. They use CI variables introduced in gitlab-org/security-products/ci-templates!298 (merged) to build, test, and release distinct images.
gemnasium:
variables:
# registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/tmp
TMP_IMAGE_PATH: "/tmp"
# registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium
# IMAGE_PATH: keep default value
# registry.gitlab.com/security-products/gemnasium
# SEC_REGISTRY: keep default value
gemnasium-maven:
variables:
# registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/tmp/maven
TMP_IMAGE_PATH: "/tmp/maven"
# registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/maven
IMAGE_PATH: "/maven"
# registry.gitlab.com/security-products/gemnasium-maven
SEC_REGISTRY_IMAGE: "$SEC_REGISTRY/gemnasium-maven"
gemnasium-python:
variables:
# registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/tmp/python
TMP_IMAGE_PATH: "/tmp/python"
# registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/python
IMAGE_PATH: "/python"
# registry.gitlab.com/security-products/gemnasium-python
SEC_REGISTRY_IMAGE: "$SEC_REGISTRY/gemnasium-python"
The Container Scanning jobs (defined in docker-test.yml
) are disabled in the child pipelines because their security reports wouldn't be processed in the Rails backend. So instead, these jobs are defined in the parent pipeline, and they cover all images.
FIPS jobs need to be updated because there's no Dockerfile.fips
in the root directory, so the rules:exists
doesn't apply.
image test
jobs need to be updated, and image_spec.rb
need to move to separate directories, or be named after the image being tested.
User documentation doesn't change.
Developer documentation is updated to reflect this new workflow.
Proof of Concept
- gitlab-org/security-products/analyzers/gemnasium!294 (closed)
- gitlab-org/security-products/analyzers/gemnasium!126 (closed)
Implementation plan
-
Update gemnasium -
Set up parent pipeline in .gitlab-ci.yml
- Keep unit tests and lint jobs.
- Keep security scans, except Container Scanning.
- Remove docker jobs.
- Add child pipelines using
trigger
jobs.
-
Set up build directory for gemnasium
, and add a child pipeline for this build.- Move and adjust
Dockerfile
. - Create
.gitlab-ci.yml
for child pipeline. - Move and adjust
Dockerfile.fips
, and configurefips
jobs - Rename
spec/image_spec.rb
and configureimage test
jobs. - Add Container Scanning jobs for
gemnasium
images to parent pipeline.
- Move and adjust
-
Set up build directory for gemnasium-maven
, and add a child pipeline for this build.- Copy and adjust
Dockerfile
. - Create
.gitlab-ci.yml
for child pipeline. - Copy and adjust
Dockerfile.fips
, and configurefips
jobs - Copy
spec/image_spec.rb
and configureimage test
jobs. - Add Container Scanning jobs for
gemnasium-maven
images to parent pipeline.
- Copy and adjust
-
Set up build directory for gemnasium-python
, and add a child pipeline for this build.- Copy and adjust
Dockerfile
. - Create
.gitlab-ci.yml
for child pipeline. - Copy and adjust
Dockerfile.fips
, and configurefips
jobs - Copy
spec/image_spec.rb
and configureimage test
jobs. - Add Container Scanning jobs for
gemnasium-python
images to parent pipeline.
- Copy and adjust
-
Update README, and tell that this project now covers gemnasium
,gemnasium-maven
, andgemnasium-python
.
-
-
Update gemnasium-maven - Tell that this project is used to maintain v2, and that v3 lives in
gemnasium
.
- Tell that this project is used to maintain v2, and that v3 lives in
-
Update gemnasium-python - Tell that this project is used to maintain v2, and that v3 lives in
gemnasium
.
- Tell that this project is used to maintain v2, and that v3 lives in
-
Check user documentation - Make sure that user docs only mentions images store in the
SEC_REGISTRY
.
- Make sure that user docs only mentions images store in the
Improvements
- There's only one CHANGELOG, README, CI config, etc..
- Any change to
gemnasium
,gemnasium-maven
orgemnasium-python
can be implemented in a single MR ingemnasium
; releasing that change is as simple as merging the MR and creating a git tag ingemnasium
. - It's possible to perform full QA when working on a MR, using all 3 images and all supported test projects.
Risks
No risk has been identified, but cascading pipelines might result in an inconsistent build state when the build partially fails.
Involved components
Optional: Intended side effects
We rebuild all images even when some are not impacted by the change.
Optional: Missing test coverage
None