Inconsistent behaviour between custom docker image and running it directly
Summary
Given a
.gitlab-ci.yml
which uses a custom image atimage:
, test execution will fail in a otherwise not reproducible manner
Steps to reproduce
- See https://gitlab.com/tmaier/gitlab-ci-issue1/tree/issue-2014 for a sample application
- branch
issue-2014
based onruby:2.3-alpine
- branch
issue-2014-default
based onruby:2.3
(finding: behaviour is independent of linux distro)
- branch
- See https://gitlab.com/tmaier/gitlab-ci-issue1/pipelines?scope=branches for the pipeline output.
- Create a
Dockerfile
and a.gitlab-ci.yml
- Add step to build the docker container and to push it to the registry
- Verify the Dockerfile by running it locally. This test passes.
$ docker run --rm registry.gitlab.com/tmaier/gitlab-ci-issue1:issue-2014 bin/rake -T
- Add jobs which tests the docker container. (See
.gitlab-ci.yml
)
rake -T custom image
uses the new docker image inimage:
. It is able to pull the image from the private repo, but the test will fail. This job failsrake -T docker-in-docker
uses docker-in-docker to run the same command with the docker image. This job passesrake -T ruby image
installs the exact same packages as inDockerfile
and runs the same commands. This job passes
Actual behavior
See above. The job fails if the image has been used with
image:
, otherwise not.
Expected behavior
Test should always pass, no matter how the image is being used
Relevant logs and/or screenshots
- See https://gitlab.com/tmaier/gitlab-ci-issue1/tree/issue-2014 for a sample application
- See https://gitlab.com/tmaier/gitlab-ci-issue1/pipelines/5607693 for the pipeline output.
Error output
Running with gitlab-ci-multi-runner 1.9.0 (82714ae)
Using Docker executor with image registry.gitlab.com/tmaier/gitlab-ci-issue1:master ...
Pulling docker image postgres:9.6-alpine ...
Starting service postgres:9.6-alpine ...
Waiting for services to be up and running...
Pulling docker image registry.gitlab.com/tmaier/gitlab-ci-issue1:master ...
Running on runner-4e4528ca-project-2238770-concurrent-0 via runner-4e4528ca-machine-1483180601-ba8988c5-digital-ocean-4gb...
Skipping Git repository setup
$ cd /usr/src/app
$ bin/rake -T
bin/rake:17:in `load': cannot load such file -- /usr/local/bundle/specifications/exe/rake (LoadError)
from bin/rake:17:in `<main>'
ERROR: Build failed: exit code 1
Expected output
$ docker run -ti --rm registry.gitlab.com/tmaier/gitlab-ci-issue1:issue-2014 bin/rake -T
Unable to find image 'registry.gitlab.com/tmaier/gitlab-ci-issue1:issue-2014' locally
issue-2014: Pulling from tmaier/gitlab-ci-issue1
b7f33cc0b48e: Already exists
863b574175a6: Already exists
aa95133b5c79: Already exists
4568db71113f: Already exists
46152aeae448: Already exists
c941f43e8c96: Pull complete
10f04c31eef6: Pull complete
438003c843fa: Pull complete
d2e820c570c4: Pull complete
128ac7d103fc: Pull complete
b569177f68cd: Pull complete
5bdad6e5016e: Pull complete
Digest: sha256:cf052d8bd402a91386c8b6d7544f7fa36f1ade704673ac4d49837382b5a853a6
Status: Downloaded newer image for registry.gitlab.com/tmaier/gitlab-ci-issue1:issue-2014
rake my_custom_task # My task... When this is shown, $ rake -T worked
Relevant is rake my_custom_task # My task... When this is shown, $ rake -T worked
Environment description
GitLab.com. See https://gitlab.com/tmaier/gitlab-ci-issue1
Used GitLab Runner version
GitLab.com setup
Running with gitlab-ci-multi-runner 1.9.0 (82714ae8)
What we know so far
- The image used here is fairly simple. It has no inherent issues by itself.
- It is independent of which ruby image is being used. The Linux distribution has no effect
- The directory
/usr/local/bundle/gems/rake-12.0.0/exe
is only empty when the image is being used in GitLab CI. Something scrubs the files