Downstream CNG pipeline build time improvements
Context
Current implementation of CNG downstream pipeline is not very cache efficient when building docker images via triggering CNG downstream pipeline from gitlab
project merge request pipelines. From the log outputs it can be seen, that even changes in gitlab
project not related to app itself, still trigger build steps of application images. Investigate if docker caching can be leveraged to reduce amount of unnecessary builds.
Note: because review-app
also uses CNG downstream pipeline, additional benefits come from this improvement even outside of E2E testing.
Implementation Plan
-
Familiarize with pipeline setup of CNG pipeline -
Investigate the reason for docker cache being invalidated -
If possible, propose/implement improvements
Areas of improvements for build time reduction
- Method that checks if image needs building is broken so all jobs are always forced to rebuild
- Image tagging doesn't work correctly in context when pipeline is triggered from upstream. All of the images end up being tagged with
master
tag which will mess with caching because different feature branch pipelines ofgitlab
project end up rewriting each others images - Default
--cache-from
indocker build
command uses single image. Using single image doesn't support caching all stages of multistage Dockerfile, which means all stages but the last one are always built from scratch - Some build jobs use
CI_PIPELINE_STARTED_AD
as input for container versions sha calculation. This means every single pipeline trigger forces image rebuild because every pipeline generates unique version even though contents used for the build of that image do not change - Main rails image Dockerfile is not very cache efficient when building app from git
sha
as pretty much every feature branch will rebuild everything past line 113 which includes installation of gems, node_modules etc. This will happen even for merge requests that don't affect app code in any way. -
workhorse
is almost always rebuilt even without any changes to workhorse code itself. This is related to copying workhorse code not from build context that would allow docker to cache the layer when files have been unchanged but related to copying code from previously built rails image which will always change in the context of triggering builds from merge requests: https://gitlab.com/gitlab-org/build/CNG/-/blob/715a66d8acda0e2b710add1132b16c6896df6747/gitlab-workhorse/Dockerfile#L12 - Several components explicitly "bust" cache through CACHE_BUSTER build arg which forces rebuilds based on date
-
final-images-listing
job can probably be skipped when triggering CNG fromgitlab
project -
mirror
project whichgitlab-org/gitlab
uses, does not have kubernetes buildx builder support. This reduces caching capabilities. We should investigate if we can also use it for mirror project. - rails image
Dockerfile
has to always installnode
dependencies in order to compileGetText PO
files viabundle exec rake gettext:compile
. This is very similar to asset compilation itself and could yield faster build times if compiled and cached these files same way assets are cached and compiled
Post Deployment Indicators
Success Measurement
Edited by Andrejs Cunskis