Move build of gemnasium-python, gemnasium-maven to gemnasium project

Summary

To reduce maintenance cost, move the build of gemnasium-maven and gemnasium-python to gemnasium, so that this project builds all Docker images related to Gemnasium: gemnasium, gemnasium-maven, and gemnasium-python.

As a result, gemnasium-maven and gemnasium-python are no longer needed, and they are archived.

This is a follow-up to #288063 (closed) (sharing the codebase), and is blocked by it.

Proposal

Doing the following as we release Gemnasium v3:

Update the gemnasium project to build, test, and release all 3 Docker images corresponding to gemnasium, gemnasium/maven, and gemnasium/python. This is implemented using child pipelines.
Update the README of gemnasium to reflect this change, and reset the CHANGELOG.
Keep gemnasium-maven and gemnasium-python to maintain v2 branch, but update the README to indicate that v3 lives in gemnasium.

gemnasium-maven:3 and gemnasium-python:3 are published in the SEC_REGISTRY, but not in the Container Registry of gemnasium-maven and gemnasium-python.

The temporary images are gemnasium/tmp, gemnasium/maven/tmp, and gemnasium/python/tmp. These images are pushed to the Container Registry of the gemnasium project.

The gemnasium project has a parent pipeline that runs the unit tests and lint the code, and child pipelines that build, test, and release the images, using trigger jobs. It includes go.yml and upsert-git-tag.yml.

Each child pipeline corresponds to a "build directory" that contains Dockerfile, main.go, .gitlab-ci.yml, and files that are specific to the build. For instance, build/gemnasium-python contains the files needed to build the gemnasium-python image.

Child pipelines include docker.yml and docker-test.yml. They use CI variables introduced in gitlab-org/security-products/ci-templates!298 (merged) to build, test, and release distinct images.

gemnasium:
  variables:
    # registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/tmp
    TMP_IMAGE_PATH: "/tmp"
    # registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium
    # IMAGE_PATH: keep default value
    # registry.gitlab.com/security-products/gemnasium
    # SEC_REGISTRY: keep default value

gemnasium-maven:
  variables:
    # registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/tmp/maven
    TMP_IMAGE_PATH: "/tmp/maven"
    # registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/maven
    IMAGE_PATH: "/maven"
    # registry.gitlab.com/security-products/gemnasium-maven
    SEC_REGISTRY_IMAGE: "$SEC_REGISTRY/gemnasium-maven"

gemnasium-python:
  variables:
    # registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/tmp/python
    TMP_IMAGE_PATH: "/tmp/python"
    # registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium/python
    IMAGE_PATH: "/python"
    # registry.gitlab.com/security-products/gemnasium-python
    SEC_REGISTRY_IMAGE: "$SEC_REGISTRY/gemnasium-python"

The Container Scanning jobs (defined in docker-test.yml) are disabled in the child pipelines because their security reports wouldn't be processed in the Rails backend. So instead, these jobs are defined in the parent pipeline, and they cover all images.

FIPS jobs need to be updated because there's no Dockerfile.fips in the root directory, so the rules:exists doesn't apply.

image test jobs need to be updated, and image_spec.rb need to move to separate directories, or be named after the image being tested.

User documentation doesn't change.

Developer documentation is updated to reflect this new workflow.

Proof of Concept

Implementation plan

Update gemnasium
- Set up parent pipeline in .gitlab-ci.yml
  - Keep unit tests and lint jobs.
  - Keep security scans, except Container Scanning.
  - Remove docker jobs.
  - Add child pipelines using trigger jobs.
- Set up build directory for gemnasium, and add a child pipeline for this build.
  - Move and adjust Dockerfile.
  - Create .gitlab-ci.yml for child pipeline.
  - Move and adjust Dockerfile.fips, and configure fips jobs
  - Rename spec/image_spec.rb and configure image test jobs.
  - Add Container Scanning jobs for gemnasium images to parent pipeline.
- Set up build directory for gemnasium-maven, and add a child pipeline for this build.
  - Copy and adjust Dockerfile.
  - Create .gitlab-ci.yml for child pipeline.
  - Copy and adjust Dockerfile.fips, and configure fips jobs
  - Copy spec/image_spec.rb and configure image test jobs.
  - Add Container Scanning jobs for gemnasium-maven images to parent pipeline.
- Set up build directory for gemnasium-python, and add a child pipeline for this build.
  - Copy and adjust Dockerfile.
  - Create .gitlab-ci.yml for child pipeline.
  - Copy and adjust Dockerfile.fips, and configure fips jobs
  - Copy spec/image_spec.rb and configure image test jobs.
  - Add Container Scanning jobs for gemnasium-python images to parent pipeline.
- Update README, and tell that this project now covers gemnasium, gemnasium-maven, and gemnasium-python.
Update gemnasium-maven
- Tell that this project is used to maintain v2, and that v3 lives in gemnasium.
Update gemnasium-python
- Tell that this project is used to maintain v2, and that v3 lives in gemnasium.
Check user documentation
- Make sure that user docs only mentions images store in the SEC_REGISTRY.

Improvements

There's only one CHANGELOG, README, CI config, etc..
Any change to gemnasium, gemnasium-maven or gemnasium-python can be implemented in a single MR in gemnasium; releasing that change is as simple as merging the MR and creating a git tag in gemnasium.
It's possible to perform full QA when working on a MR, using all 3 images and all supported test projects.

Risks

No risk has been identified, but cascading pipelines might result in an inconsistent build state when the build partially fails.

Involved components

Optional: Intended side effects

We rebuild all images even when some are not impacted by the change.

Optional: Missing test coverage

None

Edited Jun 02, 2022 by Fabien Catteau