Skip to content

Speed up docker-build job

Tomasz Maczukin requested to merge feature/speed-up-docker-build into master

This MR adds few changes that are highly speeding up the test image building job.

What was done?

1. Dockerfile refactorization

Dockerfile was refactorized to make it as much cacheable for docker builds as it can be (look into Docker docummentation for a reference):

  • ENTRYPOINT ["/tests/bin/run"] and WORKDIR /tests were moved to top because the will be probably never changed.
  • bin/prepare script was replaced by few Dockerfile lines:

    • RUN statement (splited to a few lines to be more readable) which is installing system packages.

      System packages are updated least often.

    • COPY ./Gemfile* ./ and RUN bundle install which is installing ruby gems.

      Gems - in general - are updated more often than system packages, but leas often than source code.

    • COPY ./ ./ and RUN chmod +x ./bin/* which are storing rest of the sources in docker image.

      This is the most important part of the repository and will be changed a lot - more often than gems list and much more often than system packages.

2. Docker build cache support

Since Docker 1.10 it is impossible to use docker build cache on different machines by doing a simple docker pull of previous version of the image. Image layers don't store parent chain anymore.

Docker 1.11 introduced a solution for docker build cache sharing between different docker hosts. It may be done with a proper combination of docker save and docker load.

build-image job was extended with this approach and - in combination with a Runner's caching mechanism - we can now make usage of the docker build cache.

Please read this blog post by Runnable for more information.

Results

Please look on these two builds:

What differences can we see:

  1. Steps 1-6 of docker build are get from cache. Only steps 7 and 8 are really built in this job:

    $ ./ci/docker-helper build
Sending build context to Docker daemon 220.2 kB

Step 1 : FROM ruby:2.3
2.3: Pulling from library/ruby
Digest: sha256:6da2268270168527a72962ef1a83fff5fbdf6c0c672e8f39c4b10e027ca1e858
Status: Downloaded newer image for ruby:2.3
 ---> e0f509c5f9ee
Step 2 : ENTRYPOINT /tests/bin/run
 ---> Using cache
 ---> d74db0487b18
Step 3 : WORKDIR /tests
 ---> Using cache
 ---> 0125616850dd
Step 4 : RUN sed -i "s/httpredir.debian.org/ftp.us.debian.org/" /etc/apt/sources.list &&     apt-get update &&     apt-get install -y --force-yes libqt5webkit5-dev qt5-qmake qt5-default build-essential xvfb git &&     apt-get clean
 ---> Using cache
 ---> 2c1f71b8b450
Step 5 : COPY ./Gemfile* ./
 ---> Using cache
 ---> cc34c8219bb9
Step 6 : RUN bundle install
 ---> Using cache
 ---> 93457d7bbc08
Step 7 : COPY ./ ./
 ---> b5d197cf8552
Removing intermediate container 33ca1656aaca
Step 8 : RUN chmod +x ./bin/*
 ---> Running in 2e2836d17405
 ---> 0846716eb772
Removing intermediate container 2e2836d17405
Successfully built 0846716eb772

  2. Thanks to this we've significantly shortened build time!

    First build took 13 minutes 13 seconds to build and push docker image. Second build needed for this only 5 minutes 21 seconds. It's 60% faster!

What can we do more

We could consider to execute the build-image step only on the dedicated runner.

Currently builds are executed on shared runners (auto-scaled machines) and on the dedicated runner. What this means:

  • builds executed on the dedicated runner can't re-use cache from shared runners,
  • builds executed on shared runners can't re-use cache from the dedicated runner,
  • builds executed on shared runners need to store their cache in shared S3 cache server - it's a ~600MB ZIP archive that needs to be downloaded and uploaded each time build is executed.

If we decide to use only the dedicated runner for the build-image step then we will drop the need to download/upload the cache archive (very time consuming) and also we will always have the latest cache available.

/cc @grzesiek

Merge request reports