build docker image and share cache between stages

Summary

I've a project were during CI pipeline I need to build a docker image and build a RPM who contains this docker image. I've defined different stages on my gitlab-ci.yml:

image: my-private-registry:5000/my-ci-image:latest

# Global configuration
variables:
    DOCKER_DRIVER: overlay

services:
    - docker:dind

stages:
    - prepare
    - build
    - deploy

cache:
    key: '$CI_PIPELINE_ID'
    untracked: true
    paths:
        - .build

# Builds
docker:
    stage: prepare
    script:
        - docker build -t moscow .
        - docker save -o moscow.tar moscow
        - docker tag moscow my-private-registry:5000/moscow:$CI_BUILD_REF_SLUG
        - docker push my-private-registry:5000/moscow:$CI_BUILD_REF_SLUG

rpm:
    stage: build
    image: my-private-registry:5000/vulcain:ng
    script:
        - echo "Vulcain generate RPM"
        - vulcain-rpm
    except:
        - triggers
        - master
        - tags

# Deployments
repo:
    stage: deploy
    image: my-private-registry:5000/bifrost:ng
    script:
        - |
            if [ "$CI_PROJECT_NAMESPACE" == "mynamespace" ]; then
                echo "Bifrost deploy RPM on repo"
                deploy-to-repo
            else
                echo "Namespace not allowed to push to repo"
                echo "Deployment skipped..."
            fi
    only:
        - develop
        - /^G([0-9]+)R([0-9]+)C([0-9]+)$/

For resume:

docker job build my image and save it into a tarball
rpm job build an rpm with a dedicated docker image and the rpm specfile of this project need to have a file named moscow.tar
repo job upload the generated RPM to my repo

So I need to share a cache between stages and jobs. I also need to order jobs so I use stages for this (per example in the futur I can want to build .deb at the same time that rpm).

My problem is cache doesn't work between stages if my runner configuration use privileged mode at false but I can build my docker image and when I pass privileged mode at true I can use cache between stages but I can't build my docker image...

Here is an example of my runners configuration:

[[runners]]
  name = "runner-1"
  url = "http://my-gitlab.com"
  token = "my-token"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    cache_dir = "/cache"
    shm_size = 0
    limit = 1
  [runners.cache]

I need to share the cache between stages and build my docker images.

What is the current bug behavior?

When I setup privileged mode at false the docker image build without problems but cache doesn't work. When I setup privileged mode at true the docker image doesn't build but cache work:

Here is the output of my docker build command in CI:

level=error msg="failed to dial gRPC: unable to upgrade to h2c, received 501"

What is the expected correct behavior?

(What you should see instead)

Relevant logs and/or screenshots

When privilege = false you can see an output like this for the docker job:

$ docker build -t moscow .
...
Removing intermediate container 4674945d2d56
Successfully built 4013f8349800
Successfully tagged moscow:latest
$ docker save -o moscow.tar moscow
$ docker tag moscow my-private-registry:5000/moscow:$CI_BUILD_REF_SLUG
$ docker push my-private-registry:5000/moscow:$CI_BUILD_REF_SLUG
The push refers to a repository [docker-registry.ecocenter.fr:5000/moscow]
a2a497bb647b: Preparing
237b043d96ff: Preparing
d928667b3bbb: Preparing
9d8586ce13d6: Preparing
0c7ab3dd0440: Preparing
e154057080f4: Preparing
e154057080f4: Waiting
d928667b3bbb: Pushed
e154057080f4: Layer already exists
0c7ab3dd0440: Pushed
237b043d96ff: Pushed
a2a497bb647b: Pushed
9d8586ce13d6: Pushed
develop: digest: sha256:579a88198222f974308ff6cf24d6616bbdf028bf01e35960517442c7a9c9a45b size: 1580
Creating cache 3468...
WARNING: .build: no matching files                 
moscow.tar: found 1 matching files                 
untracked: found 1 files                           
Created cache
Job succeeded

but during the rpm job the cache seems to be not shared:

$ echo "Vulcain generate RPM"
Vulcain generate RPM
$ vulcain-rpm
---------------------------------------------------------
project name: moscow
version: 0.0.0
revision: 141.vim.develop
branch name: develop
namespace: vim
project dir: /builds/vim/moscow
---------------------------------------------------------
Need to build RPM
+ expect
++ vulcain-project-name
spawn rpmbuild -bb --sign /root/rpmbuild/SPECS/moscow.spec
+ umask 022
+ cd /root/rpmbuild/BUILD
+ '[' /root/rpmbuild/BUILDROOT/moscow-0.0.0-141.vim.develop.x86_64 '!=' / ']'
+ rm -rf /root/rpmbuild/BUILDROOT/moscow-0.0.0-141.vim.develop.x86_64
++ dirname /root/rpmbuild/BUILDROOT/moscow-0.0.0-141.vim.develop.x86_64
+ mkdir -p /root/rpmbuild/BUILDROOT
+ mkdir /root/rpmbuild/BUILDROOT/moscow-0.0.0-141.vim.develop.x86_64
+ mkdir -p /root/rpmbuild/BUILDROOT/moscow-0.0.0-141.vim.develop.x86_64//usr/share/ecv/moscow
+ cp /root/rpmbuild/SOURCES/moscow.tar /root/rpmbuild/BUILDROOT/moscow-0.0.0-141.vim.develop.x86_64//usr/share/ecv/moscow/
cp: cannot stat '/root/rpmbuild/SOURCES/moscow.tar': No such file or directory

Another strange behavior is that when I setup privileged = false I obtain this warning on each jobs:

Running with gitlab-ci-multi-runner 9.5.0 (413da38)
  on makeanix-8 (0f9e62b2)
Using Docker executor with image my-private-registry:5000/vulcain:ng ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image docker:dind ID=sha256:72904c4aeb8eb51b36ccba92c543e01ee4bb9459f95270e1fc4a4609f7e0e52a for docker service...
Waiting for services to be up and running...

*** WARNING: Service runner-0f9e62b2-project-244-concurrent-0-docker-0 probably didn't start properly.

Error response from daemon: Cannot link to a non running container: /runner-0f9e62b2-project-244-concurrent-0-docker-0 AS /runner-0f9e62b2-project-244-concurrent-0-docker-0-wait-for-service/runner-0f9e62b2-project-244-concurrent-0-docker-0

2018-01-11T16:03:51.443751703Z mount: permission denied (are you root?)
2018-01-11T16:03:51.443811356Z Could not mount /sys/kernel/security.
2018-01-11T16:03:51.444905498Z AppArmor detection and --privileged mode might break.
2018-01-11T16:03:51.447567588Z mount: permission denied (are you root?)

When I setup privileged = true this error disapear.

And sometimes without made changes the cache was shared only between 2 jobs but never on each jobs... I really don't understand what's happening

Results of GitLab environment info

$ gitlab-ci-multi-runner --version
Version:      9.5.0
Git revision: 413da38
Git branch:   9-5-stable
GO version:   go1.8.3
Built:        Tue, 22 Aug 2017 13:35:13 +0000
OS/Arch:      linux/amd64

And gitlab informations:

GitLab 10.0.1
GitLab Shell 5.9.0
GitLab Workhorse v3.0.0
GitLab API v4
Git 2.13.5
Ruby 2.3.5p376
Rails 4.2.8
postgresql 9.6.3

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Additional informations

I've try different solutions like this => https://gitlab.com/gitlab-org/gitlab-ce/issues/17769 but unfortunately it doesn't work

Edited Jan 11, 2018 by hervé beraud