Skip to content
  • Steve Xuereb's avatar
    Use docker volumes instead of cache containers · ac8efb11
    Steve Xuereb authored
    Problem:
    `cache containers` are containers created specifically to create volumes
    and initialize them correctly and exit properly. Then each container that is created
    afterword the `--volumes-from` is passed specifying the container ids of
    the **exited** cache containers. As seen in
    https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450#note_316034514
    the
    [`ContainerCreate`](https://docs.docker.com/engine/api/v1.25/#operation/ContainerCreate)
    can return a 404 if we define a `--volumes-from` to a nonexisting
    container. Exited containers can easily be removed/cleanup by some other
    system they are also susceptible to oom killer/cgroups so the resource
    is more temporary.
    
    Solution:
    Instead of creating a cache container and then keep reusing that
    container to specify volume mounts. Create a volume with `docker volume
    create` and define it part of the bindings, for example
    `$VOLUME_NAME:/cache`. Using volumes directly has the following
    benefits:
    - `gitlab.com/gitlab-org/gitlab-runner/executors/docker/internal/volumes`
      is much simpler since we don't have to manage more containers.
    - Volumes don't randomly get removed the same way as exited cache
      containers do.
    - Permissions are set by default since `rw` is implied, so no
      [`cache-init`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/ad6cd05d/commands/helpers/cache_init.go)
      command is needed.
    - If a volume with that name is already created it's reused, Docker
      doesn't return an error.
    - Achieve what we are doing in fewer steps, creating fewer containers
      which makes the build faster especially on Windows where we used large
      containers.
    
    The volumes aren't labeled the same as the current ones that we create,
    for this please take a look at
    https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25300.
    
    Testing:
    https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1989#testing
    
    Docker commands we used to run:
    
    ```
    docker inspect $CACHE_CONTAINER
    docker create gitlab/gitlab-runner-helper:xxxx cache-init
    docker start $CACHE_CONTAINER
    docker wait $CACHE_CONTAINER
    docker run --volumes-from $CACHE_CONTAINER $IMAGE
    ```
    
    Docker commands we run now:
    
    ```
    docker volume create $VOLUME_NAME
    docker run -v $VOLUME_NAME:/cache $IMAGE
    ```
    
    reference https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450
    ac8efb11