Use caches to further reduce GDK E2E docker image size and build time
As part of gitlab-org/quality/quality-engineering&4 (closed) we're runing end-to-end (QA) tests on GDK in MRs and in master pipelines.
!117622 (merged) improves on the initial implementation by reducing the size of the docker images we use, and speeds up overall time to build and run tests, but building still takes too long (~25 mins) and leaves images too large (~11GB).
This issue proposes some options for further reducing the image size and job times.
Tasks
-
Consider if any more caches can be saved outside the GDK container and shared between pipelines. For example, as in gitlab-development-kit!2799 (merged)Docker has a very powerful built-in caching mechanisms. Trying to cache something outside of build to use during build runtime is rather hacky and probably isn't worth it. -
Consider being not rebuilding the 2nd image stage within the same MR if no code changes between pipelines (e.g., if a commit only adds documentation, or a new pipeline is triggered without any changes)Optimise dockerfile to reuse docker caching as much as possible and rebuild layers only when relevant files change -
Remove building separate base image as it can lead to unpredictable results if breaking changes are introduced in master branch. Have only one single gdk image and rely on docker caching to intelligently rebuild parts that have changed files -
Allow skipping gitaly, workhorse and gitlab-shell setup when running make all
task because we build them in separate previous docker layers
Edited by Andrejs Cunskis