Docker Cache Invalidation
Summary
I am experiencing what I feel is inconsistent behavior using the remote docker build cache pattern in Gitlab CI. I initially raised this issue via Gitlab support, and I have been directed to create an issue here. The tl;dr is that the remote docker cache appears to be invalidated in a situation which it should not be. This triggers a full re-build, which has a big impact on our development speed.
Steps to reproduce
- Create a new repository (excluding
.gitlab-ci.yml
for now) - Add a Dockerfile with the following contents:
FROM ubuntu:18.04
RUN echo 'long process'
COPY code /code/
- Add a directory at the root called
code/
with some file in it (examplecode/helloworld.py
) with some code in it
print('hello world')
- Commit/Push these changes to master
- Create a new branch locally, and add a
.gitlab-ci.yml
file with the following:
# GitLab CI steps.
stages:
- build
variables:
LATEST_TEST_IMAGE_TAG: $CI_REGISTRY_IMAGE/test:latest
TEST_IMAGE_BRANCH_TAG: $CI_REGISTRY_IMAGE/test:$CI_COMMIT_REF_SLUG
DOCKER_HOST: tcp://docker:2375
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
DOCKER_BUILDKIT: 1
build:
image: docker:git
stage: build
services:
- docker:dind
script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
- docker build --tag $TEST_IMAGE_BRANCH_TAG
--cache-from $TEST_IMAGE_BRANCH_TAG
--build-arg BUILDKIT_INLINE_CACHE=1 "."
- docker push $TEST_IMAGE_BRANCH_TAG
- Commit and push this
- Confirm that the build completes successfully in CI and inspect the build logs, since it is the first build you should see
RUN echo 'long process'
being run. - In the same branch, update the code in
code/helloworld.py
, commit and push - Inspect the build logs, you should see the
ECHO
line being cached, but theCOPY
line needing to be run (which is correct since those files changed) - Make another change to the code, commit and push
- Inspect the logs and see that echo
ECHO
line is no longer cached, the full build runs again - Repeat as desired to observe cyclical behavior
Actual behavior
The second change to code/
triggers a full rebuild
Expected behavior
The second change to code/
should use the cache (like the first one) but it does not.
Relevant logs and/or screenshots
Here are the build logs from a test example I did
First build
Running with gitlab-runner 11.7.0 (8bb608ff)
on ip-########
Using Docker executor with image docker:git ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:66dc2d45749a48592f4348fb3d567bdd65c9dbd5402a413b6d169619e32f6bd2 for docker:dind ...
Waiting for services to be up and running...
Pulling docker image docker:git ...
Using docker image sha256:3a94f70af8c72bde9aff3c4ccd7b8d293028223ee479a820e1c8e97e139bf513 for docker:git ...
Running on runner-######### via ip-#########..
Cloning repository...
Cloning into '#################'...
Checking out 64e30c7a as ci...
Skipping Git submodules setup
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build --tag $TEST_IMAGE_BRANCH_TAG --cache-from $TEST_IMAGE_BRANCH_TAG --build-arg BUILDKIT_INLINE_CACHE=1 "."
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 98B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/ubuntu:18.04
#3 DONE 0.2s
#6 [internal] load build context
#6 DONE 0.0s
#4 [1/3] FROM docker.io/library/ubuntu:18.04@sha256:05a58ded9a2c792598e8f4a...
#4 DONE 0.0s
#8 importing cache manifest from ########################...
#8 ERROR: ##################### not found
#6 [internal] load build context
#6 transferring context: 115B done
#6 DONE 0.0s
#4 [1/3] FROM docker.io/library/ubuntu:18.04@sha256:05a58ded9a2c792598e8f4a...
#4 resolve docker.io/library/ubuntu:18.04@sha256:05a58ded9a2c792598e8f4aa8ffe300318eac6f294bf4f49a7abae7544918592 done
#4 sha256:3baa9cb2483bd9c5329a44d9c2fe72535625bbd4308bca95785dd58e72c06365 35.36kB / 35.36kB 0.1s done
#4 sha256:94e5ff4c0b1526abf77c236655f21c8f67a23313291c8b970fe6b469549d8153 848B / 848B 0.1s done
#4 sha256:05a58ded9a2c792598e8f4aa8ffe300318eac6f294bf4f49a7abae7544918592 1.42kB / 1.42kB done
#4 sha256:d1bf40f712c466317f5e06d38b3e7e4c98fef1229872bf6e2a8d1e01836c7ec4 1.15kB / 1.15kB done
#4 sha256:6526a1858e5d72cdffdcb41874f5f7ab3e7d531836a45bb144bf650b7242ce20 3.41kB / 3.41kB done
#4 sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 747.03kB / 26.70MB 0.1s
#4 sha256:1860925334f940c3145808527480b4f0cba7f01279087fdb27679e4354fba967 163B / 163B 0.1s done
#4 sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 11.64MB / 26.70MB 0.2s
#4 sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 22.12MB / 26.70MB 0.3s
#4 sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 26.70MB / 26.70MB 0.4s done
#4 extracting sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43
#4 extracting sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 0.8s done
#4 extracting sha256:3baa9cb2483bd9c5329a44d9c2fe72535625bbd4308bca95785dd58e72c06365 done
#4 extracting sha256:94e5ff4c0b1526abf77c236655f21c8f67a23313291c8b970fe6b469549d8153
#4 extracting sha256:94e5ff4c0b1526abf77c236655f21c8f67a23313291c8b970fe6b469549d8153 done
#4 extracting sha256:1860925334f940c3145808527480b4f0cba7f01279087fdb27679e4354fba967 done
#4 DONE 1.7s
#5 [2/3] RUN echo 'long process'
#5 0.286 long process
#5 DONE 0.4s
#7 [3/3] COPY code /code/
#7 DONE 0.0s
#9 exporting to image
#9 exporting layers done
#9 writing image sha256:3de48a4fa457f872d7bd749e6717bd559e5a2cc4a5b15a860e1f6fa580c01c53 done
#9 naming to #############################
#9 DONE 0.0s
#10 exporting cache
#10 preparing build cache for export done
#10 DONE 0.0s
------
> importing cache manifest from #############################
------
docker push $TEST_IMAGE_BRANCH_TAG
The push refers to repository [#############################]
53504f37851d: Preparing
5f70bf18a086: Preparing
001e4a80973b: Preparing
2ba5b91ca2b0: Preparing
2f37d1102187: Preparing
79bde4d54386: Preparing
79bde4d54386: Waiting
2f37d1102187: Pushed
001e4a80973b: Pushed
5f70bf18a086: Pushed
2ba5b91ca2b0: Pushed
53504f37851d: Pushed
79bde4d54386: Pushed
ci: digest: sha256:38421d04bbc7c69f7495c053fa424a3d2263403c1259aefea50fd90ea348284b size: 1565
Job succeeded
Second build (cache works)
Running with gitlab-runner 11.7.0 (8bb608ff)
on ip-#################
Using Docker executor with image docker:git ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:66dc2d45749a48592f4348fb3d567bdd65c9dbd5402a413b6d169619e32f6bd2 for docker:dind ...
Waiting for services to be up and running...
Pulling docker image docker:git ...
Using docker image sha256:3a94f70af8c72bde9aff3c4ccd7b8d293028223ee479a820e1c8e97e139bf513 for docker:git ...
Running on runner-############ via ip-#######...
Fetching changes...
HEAD is now at 64e30c7 add CI
From ##################3
64e30c7..3cef1c0 ci -> origin/ci
Checking out 3cef1c09 as ci...
Skipping Git submodules setup
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build --tag $TEST_IMAGE_BRANCH_TAG --cache-from $TEST_IMAGE_BRANCH_TAG --build-arg BUILDKIT_INLINE_CACHE=1 "."
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 98B done
#1 DONE 0.0s
#3 [internal] load metadata for docker.io/library/ubuntu:18.04
#3 DONE 0.2s
#7 [internal] load build context
#7 DONE 0.0s
#4 [1/3] FROM docker.io/library/ubuntu:18.04@sha256:05a58ded9a2c792598e8f4a...
#4 DONE 0.0s
#5 importing cache manifest from #####################3
#5 DONE 0.6s
#7 [internal] load build context
#7 transferring context: 129B done
#7 DONE 0.0s
#6 [2/3] RUN echo 'long process'
#6 pulling sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43
#6 pulling sha256:3baa9cb2483bd9c5329a44d9c2fe72535625bbd4308bca95785dd58e72c06365
#6 pulling sha256:94e5ff4c0b1526abf77c236655f21c8f67a23313291c8b970fe6b469549d8153
#6 pulling sha256:3baa9cb2483bd9c5329a44d9c2fe72535625bbd4308bca95785dd58e72c06365 0.2s done
#6 pulling sha256:1860925334f940c3145808527480b4f0cba7f01279087fdb27679e4354fba967
#6 pulling sha256:94e5ff4c0b1526abf77c236655f21c8f67a23313291c8b970fe6b469549d8153 0.2s done
#6 pulling sha256:1860925334f940c3145808527480b4f0cba7f01279087fdb27679e4354fba967 0.1s done
#6 pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#6 pulling sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 0.4s done
#6 pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.1s done
#6 CACHED
#8 [3/3] COPY code /code/
#8 DONE 0.0s
#9 exporting to image
#9 exporting layers done
#9 writing image sha256:19349817088abc5038e4d23eb2e0554db9a008884935599e7a45292364e7d16b done
#9 naming to ################### done
#9 DONE 0.0s
#10 exporting cache
#10 preparing build cache for export done
#10 DONE 0.0s
$ docker push $TEST_IMAGE_BRANCH_TAG
The push refers to repository [###################]
06ad1e41b8a8: Preparing
5f70bf18a086: Preparing
001e4a80973b: Preparing
2ba5b91ca2b0: Preparing
2f37d1102187: Preparing
79bde4d54386: Preparing
79bde4d54386: Waiting
2f37d1102187: Layer already exists
001e4a80973b: Layer already exists
5f70bf18a086: Layer already exists
79bde4d54386: Layer already exists
2ba5b91ca2b0: Layer already exists
06ad1e41b8a8: Pushed
ci: digest: sha256:4a3d8f29a086423b3ad22a3f001eafbe5ae979f3a4b9cfb352ac7ff3db0136bd size: 1565
Job succeeded
And finally the third (no cache)
Running with gitlab-runner 11.7.0 (8bb608ff)
on ip-####################
Using Docker executor with image docker:git ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:66dc2d45749a48592f4348fb3d567bdd65c9dbd5402a413b6d169619e32f6bd2 for docker:dind ...
Waiting for services to be up and running...
Pulling docker image docker:git ...
Using docker image sha256:3a94f70af8c72bde9aff3c4ccd7b8d293028223ee479a820e1c8e97e139bf513 for docker:git ...
Running on runner-############## via ip-#################### ...
Fetching changes...
HEAD is now at 3cef1c0 now change tests
From ###################
3cef1c0..1e69e71 ci -> origin/ci
Checking out 1e69e71b as ci...
Skipping Git submodules setup
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build --tag $TEST_IMAGE_BRANCH_TAG --cache-from $TEST_IMAGE_BRANCH_TAG --build-arg BUILDKIT_INLINE_CACHE=1 "."
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 98B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/ubuntu:18.04
#3 DONE 0.2s
#7 [internal] load build context
#7 DONE 0.0s
#5 [1/3] FROM docker.io/library/ubuntu:18.04@sha256:05a58ded9a2c792598e8f4a...
#5 DONE 0.0s
#4 importing cache manifest from #####################
#4 DONE 0.8s
#7 [internal] load build context
#7 transferring context: 136B done
#7 DONE 0.0s
#5 [1/3] FROM docker.io/library/ubuntu:18.04@sha256:05a58ded9a2c792598e8f4a...
#5 resolve docker.io/library/ubuntu:18.04@sha256:05a58ded9a2c792598e8f4aa8ffe300318eac6f294bf4f49a7abae7544918592 done
#5 sha256:3baa9cb2483bd9c5329a44d9c2fe72535625bbd4308bca95785dd58e72c06365 0B / 35.36kB 0.1s
#5 sha256:94e5ff4c0b1526abf77c236655f21c8f67a23313291c8b970fe6b469549d8153 848B / 848B 0.1s done
#5 sha256:1860925334f940c3145808527480b4f0cba7f01279087fdb27679e4354fba967 0B / 163B 0.1s
#5 sha256:05a58ded9a2c792598e8f4aa8ffe300318eac6f294bf4f49a7abae7544918592 1.42kB / 1.42kB done
#5 sha256:d1bf40f712c466317f5e06d38b3e7e4c98fef1229872bf6e2a8d1e01836c7ec4 1.15kB / 1.15kB done
#5 sha256:6526a1858e5d72cdffdcb41874f5f7ab3e7d531836a45bb144bf650b7242ce20 3.41kB / 3.41kB done
#5 sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 0B / 26.70MB 0.1s
#5 sha256:3baa9cb2483bd9c5329a44d9c2fe72535625bbd4308bca95785dd58e72c06365 35.36kB / 35.36kB 0.2s done
#5 sha256:1860925334f940c3145808527480b4f0cba7f01279087fdb27679e4354fba967 163B / 163B 0.1s done
#5 sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 12.01MB / 26.70MB 0.4s
#5 sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 19.35MB / 26.70MB 0.5s
#5 sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 26.70MB / 26.70MB 0.7s
#5 sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 26.70MB / 26.70MB 0.7s done
#5 extracting sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 0.1s
#5 extracting sha256:f08d8e2a3ba11bea23cf5c17e8e1c620057412ed05c32d1114640e18d6dd0a43 0.8s done
#5 extracting sha256:3baa9cb2483bd9c5329a44d9c2fe72535625bbd4308bca95785dd58e72c06365 done
#5 extracting sha256:94e5ff4c0b1526abf77c236655f21c8f67a23313291c8b970fe6b469549d8153 done
#5 extracting sha256:1860925334f940c3145808527480b4f0cba7f01279087fdb27679e4354fba967 done
#5 DONE 2.0s
#6 [2/3] RUN echo 'long process'
#6 0.258 long process
#6 DONE 0.3s
#8 [3/3] COPY code /code/
#8 DONE 0.0s
#9 exporting to image
#9 exporting layers done
#9 writing image sha256:809be1b2979a2c1485d90e6d7d4cd616b240c61f80f42b2071011cea4fdbd550 done
#9 naming to ##################### done
#9 DONE 0.0s
#10 exporting cache
#10 preparing build cache for export done
#10 DONE 0.0s
$ docker push $TEST_IMAGE_BRANCH_TAG
The push refers to repository [#####################]
9634b8386180: Preparing
5f70bf18a086: Preparing
001e4a80973b: Preparing
2ba5b91ca2b0: Preparing
2f37d1102187: Preparing
79bde4d54386: Preparing
79bde4d54386: Waiting
2ba5b91ca2b0: Layer already exists
2f37d1102187: Layer already exists
001e4a80973b: Layer already exists
79bde4d54386: Layer already exists
9634b8386180: Pushed
5f70bf18a086: Pushed
ci: digest: sha256:1d473fa33f744301ec85c01de4b2f9659826dd87cbc9ca1445fcb9368d1b6e95 size: 1565
Job succeeded
Environment description
We are using our own runner (on EC2):
concurrent = 4
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "ip-##########"
url = "https://gitlab.com/"
token = "############"
executor = "docker"
[runners.docker]
tls_verify = true
image = "ubuntu:18.04"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Used GitLab Runner version
See build logs above
Possible fixes
None that I could find