Codequality with dind on rootless docker runner

Proposal

Currently, for rootless docker runner setups, we are suggesting (in this doc: Run Code Quality rootless with private runners) that customer share /run/user/<gitlab-runner-user>/docker.sock inside their build container and mount it when docker run script is executed. Some customers don't want to share the docker.sock of the host and want to use dind service instead.

It has proven to be not a trivial thing to use dind service on rootless docker and when customers try it, they are receiving the following error:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF: unknown

I managed to replicate it using the following configuration:

config.toml

To achieve this configuration, I followed Run the Docker daemon as a non-root user (Rootless mode) tutorial:

concurrent = 1
check_interval = 0
log_level = "debug"
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "instance-20240229-141025.c.some-user.internal"
  url = "https://gitlab.com"
  id = 38910105
  token = "TOKEN"
  token_obtained_at = 2024-06-26T09:43:53Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "ruby:2.7"
    privileged = false
    services_privileged = true
    allowed_privileged_services = ['docker.io/library/docker:*-dind', 'docker:*-dind', 'docker.io/library/docker:dind', 'docker:dind']
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0
    host = "unix:///run/user/USER_ID/docker.sock"

.gitlab-ci.yml

include:
  - template: Jobs/Code-Quality.gitlab-ci.yml

Job log

Running with gitlab-runner 16.9.1 (782c6ecb)
  on instance-20240229-141025.c.khrechyshkina-60eb6363.internal 3_dEcQ-8n, system ID: s_03ec4870fd13
section_start:1719397770:resolve_secrets
Resolving secrets
section_end:1719397770:resolve_secrets
section_start:1719397770:prepare_executor
Preparing the "docker" executor
Using Docker executor with image docker:20.10.12 ...
Starting service docker:20.10.12-dind ...
Pulling docker image docker:20.10.12-dind ...
Using docker image sha256:1a42336ff683d7dadd320ea6fe9d93a5b101474346302d23f96c9b4546cb414d for docker:20.10.12-dind with digest docker@sha256:6f2ae4a5fd85ccf85cdd829057a34ace894d25d544e5e4d9f2e7109297fedf8d ...
Waiting for services to be up and running (timeout 30 seconds)...
Pulling docker image docker:20.10.12 ...
Using docker image sha256:15a9bc7c6340df2ac9d6c8196ca1d905180ddf2ca8b29a8d98f5422e2e5ccf85 for docker:20.10.12 with digest docker@sha256:a729cce205a05b0b86dc8dca87823efaffc3f74979fe7dc86a707c2fbf631b61 ...
section_end:1719397775:prepare_executor
section_start:1719397775:prepare_script
Preparing environment
Running on runner-3decq-8n-project-59313658-concurrent-0 via instance-20240229-141025.c.khrechyshkina-60eb6363.internal...
section_end:1719397775:prepare_script
section_start:1719397775:get_sources
Getting source from Git repository
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/gl-demo-ultimate-khrechyshkina/tickets/zd543202/.git/
Checking out 4db7325e as detached HEAD (ref is main)...

Skipping Git submodules setup
section_end:1719397777:get_sources
section_start:1719397777:step_script
Executing "step_script" stage of the job script
Using docker image sha256:15a9bc7c6340df2ac9d6c8196ca1d905180ddf2ca8b29a8d98f5422e2e5ccf85 for docker:20.10.12 with digest docker@sha256:a729cce205a05b0b86dc8dca87823efaffc3f74979fe7dc86a707c2fbf631b61 ...
$ export SOURCE_CODE=${SOURCE_CODE:-$PWD}
$ if ! docker info &>/dev/null; then # collapsed multi-line command
$ function propagate_env_vars() { # collapsed multi-line command
$ if [ -n "$CODECLIMATE_REGISTRY_USERNAME" ] && [ -n "$CODECLIMATE_REGISTRY_PASSWORD" ] && [ -n "$CODECLIMATE_PREFIX" ]; then # collapsed multi-line command
$ docker pull --quiet "$CODE_QUALITY_IMAGE"
registry.gitlab.com/gitlab-org/ci-cd/codequality:0.96.0
$ docker run --rm \ # collapsed multi-line command
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF: unknown.
section_end:1719397787:step_script
section_start:1719397787:upload_artifacts_on_failure
Uploading artifacts for failed job
Uploading artifacts...
WARNING: gl-code-quality-report.json: no matching files. Ensure that the artifact path is relative to the working directory (/builds/gl-demo-ultimate-khrechyshkina/tickets/zd543202) 
ERROR: No files to upload                          
section_end:1719397788:upload_artifacts_on_failure
section_start:1719397788:cleanup_file_variables
Cleaning up project directory and file based variables
section_end:1719397789:cleanup_file_variables
ERROR: Job failed: exit code 125


It is not clear whether it is possible to achieve Codequality on rootless docker runner without sharing the host's docker socket inside the container, but there is a demand for it, see this internal customer ticket.