Pulling from Docker takes a long time - local images add build cost 5min cost each
Hi,
related: #225 (closed) #149 (closed) and Docker https://forums.docker.com/t/very-slow-pull-download-times/999
we reverted to 0.5.5.1, because since 0.6, the image refresh / expiration was introduced. So now, almost all builds will:
- check, if our local, custom docker image (with deployment keys, etc) is available on Docker registry (which is not), and then fall back to the local -> takes 5min for each build step
- checks postgres, elasticsearch, redis, (which might be ok, as it is about 2-3seconds if Docker is available & responsive), as the 1 minute expiration is quite low
A build, consisting of 3 build steps, which normally takes like 5 min in total, will now take 5+(3*5) = 20 minutes!
gitlab-ci-multi-runner 0.6.2 (3227f0a)
Using Docker executor with image pludoni/rails-base:v1 ...
Pulling docker image pludoni/rails-base:v1 ...
WARNING: Cannot pull the latest version of image pludoni/rails-base:v1 : Error: image pludoni/rails-base:v1 not found
WARNING: Locally found image will be used instead.
Pulling docker image postgres:9.4 ...
Starting service postgres:9.4 ...
Pulling docker image redis:latest ...
Starting service redis:latest ...
Waiting for services to be up and running...
Pulling docker image gitlab/gitlab-runner:service ...
Pulling docker image gitlab/gitlab-runner:service ...
Running on runner-fe51a2a8-project-2-concurrent-0 via hal.pludoni...
Investigating further, the problem lies in docker pull, if the image is not there:
(run on a 32gb, 8core machine, Ubuntu 12.04, 50mbit internet connection)
$ docker --version
Docker version 1.9.0, build 76d6bc9
# pulling image, that's already there
$ time docker pull postgres:9.4
9.4: Pulling from library/postgres
Digest: sha256:e784c88c976d4f1f391663163c3b7ddb99064a78d43ea82d5612d9224dd9a532
Status: Image is up to date for postgres:9.4
docker pull postgres:9.4
0,04s user 0,01s system 1% cpu 2,547 total
# 2.5 seconds, not too bad
# pulling image, that's not on Docker
$ time docker pull pludoni/rails-base:v1
Pulling repository docker.io/pludoni/rails-base
Error: image pludoni/rails-base:v1 not found
docker pull pludoni/rails-base:v1 0,07s user 0,01s system 0% cpu 5:06,10 total
That's >5 minutes for querying that image. :(
0.5 works great for us, so is there any chance to speed up things? I mean, except hosting a private Docker registry.
In general, pinging Docker every 1 minute seems also unnecessary, as there might be DNS/network failures or docker might be slow or down.
1.) Could we make the 1 minute TTL for images configurable?
2.) Could we somehow speed up the use of local images? Perhaps a config flag or a timeout for docker pull?
Otherwise, I would be good to mention it in the installation documentation, when the user want to use local images, that they have to run a private Docker registry.
thanks for your time!