Draft: Build, test, push images using parent-child pipelines (PoC)
What does this MR do?
Use parent-child pipelines to build, test, and push 2 different images: gemnasium and gemnasium/python. This PoC is conducted in the context of gitlab-org/gitlab#198361 (closed)
The parent pipeline repeats the pre-build stage defined in the analyzer.yml template. It's not possible to include this template because it declares other stages that are executed in the child pipelines, and these cannot be removed.
The child pipelines have specific CI configuration files, like build/gemnasium-python/.gitlab-ci.yml. These include analyzer.yml but all jobs of the pre-build stage have been disabled. Also, the script of the build tmp image job has been changed so that the Dockerfile can be set to build/gemnasium/Dockerfile and build/gemnasiumt-python/Dockerfile.
Dockerfiles must be multi-stage Dockefiles because what's compiled in the pre-build stage (parent pipeline) cannot be past to the build stage (child pipeline) as a job artifact. Hopefully this change is aligned with gitlab-org/gitlab#9736 (closed).
The child pipeline cannot build gemnasium-python because it doesn't have write access to the gemnasium-python project, so it builds gemnasium/python instead. Next step is to trigger a pipeline in the gemnasium-python project to re-publish gemnasium/python as gemnasium-python, for backward compatibility.
What has been proven?
Using parent-child pipelines, the Gemnasium project can build all Gemnasium-based images. The CI configurations and Dockerfiles are easy to maintain, and the pipeline isn't too complex.
What are the limitations?
The shared CI template analyzer.yml cannot be reused. The pre-build stage needs to be repeated in the CI config of the parent pipeline. Other stages can be used in the child pipelines, but this isn't clean, and might be error prone.
Security scanners are repeated across the child pipelines, and their reports are ignored (empty Security tab in the pipeline page). They should move to the parent pipeline.
What needs to be further explored?
make the child pipeline that creates gemnasium/python trigger a pipeline in gemnasium-python (external project), in order to republish gemnasium/python as gemnasium-python
Other takeaways
Currently the share CI config analyzer.yml makes it really difficult to extract what's specific to Docker images (build, test, push) to a child pipeline, and keep everything else in the parent pipeline.
If Dependency Scanning is disabled in a parent pipeline using DEPENDENCY_SCANNING_DISABLED variable, then it cannot be restored in its child pipelines using the same variable. That's because the rules of the dependency scanning jobs check for when the variable is set, and it cannot be unset. Same applies to SAST and other Secure analyzers.