Codequality with dind on rootless docker runner

Proposal

Currently, for rootless docker runner setups, we are suggesting (in this doc: Run Code Quality rootless with private runners) that customer share /run/user/<gitlab-runner-user>/docker.sock inside their build container and mount it when docker run script is executed. Some customers don't want to share the docker.sock of the host and want to use dind service instead.

It has proven to be not a trivial thing to use dind service on rootless docker and when customers try it, they are receiving the following error:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF: unknown

I managed to replicate it using the following configuration:

config.toml

To achieve this configuration, I followed Run the Docker daemon as a non-root user (Rootless mode) tutorial:

concurrent = 1
check_interval = 0
log_level = "debug"
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "instance-20240229-141025.c.some-user.internal"
  url = "https://gitlab.com"
  id = 38910105
  token = "TOKEN"
  token_obtained_at = 2024-06-26T09:43:53Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "ruby:2.7"
    privileged = false
    services_privileged = true
    allowed_privileged_services = ['docker.io/library/docker:*-dind', 'docker:*-dind', 'docker.io/library/docker:dind', 'docker:dind']
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0
    host = "unix:///run/user/USER_ID/docker.sock"

.gitlab-ci.yml

include:
  - template: Jobs/Code-Quality.gitlab-ci.yml

Job log

[0KRunning with gitlab-runner 16.9.1 (782c6ecb)[0;m
[0K  on instance-20240229-141025.c.khrechyshkina-60eb6363.internal 3_dEcQ-8n, system ID: s_03ec4870fd13[0;m
section_start:1719397770:resolve_secrets
[0K[0K[36;1mResolving secrets[0;m[0;m
section_end:1719397770:resolve_secrets
[0Ksection_start:1719397770:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m[0;m
[0KUsing Docker executor with image docker:20.10.12 ...[0;m
[0KStarting service docker:20.10.12-dind ...[0;m
[0KPulling docker image docker:20.10.12-dind ...[0;m
[0KUsing docker image sha256:1a42336ff683d7dadd320ea6fe9d93a5b101474346302d23f96c9b4546cb414d for docker:20.10.12-dind with digest docker@sha256:6f2ae4a5fd85ccf85cdd829057a34ace894d25d544e5e4d9f2e7109297fedf8d ...[0;m
[0KWaiting for services to be up and running (timeout 30 seconds)...[0;m
[0KPulling docker image docker:20.10.12 ...[0;m
[0KUsing docker image sha256:15a9bc7c6340df2ac9d6c8196ca1d905180ddf2ca8b29a8d98f5422e2e5ccf85 for docker:20.10.12 with digest docker@sha256:a729cce205a05b0b86dc8dca87823efaffc3f74979fe7dc86a707c2fbf631b61 ...[0;m
section_end:1719397775:prepare_executor
[0Ksection_start:1719397775:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
Running on runner-3decq-8n-project-59313658-concurrent-0 via instance-20240229-141025.c.khrechyshkina-60eb6363.internal...
section_end:1719397775:prepare_script
[0Ksection_start:1719397775:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m[0;m
[32;1mFetching changes with git depth set to 20...[0;m
Reinitialized existing Git repository in /builds/gl-demo-ultimate-khrechyshkina/tickets/zd543202/.git/
[32;1mChecking out 4db7325e as detached HEAD (ref is main)...[0;m

[32;1mSkipping Git submodules setup[0;m
section_end:1719397777:get_sources
[0Ksection_start:1719397777:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
[0KUsing docker image sha256:15a9bc7c6340df2ac9d6c8196ca1d905180ddf2ca8b29a8d98f5422e2e5ccf85 for docker:20.10.12 with digest docker@sha256:a729cce205a05b0b86dc8dca87823efaffc3f74979fe7dc86a707c2fbf631b61 ...[0;m
[32;1m$ export SOURCE_CODE=${SOURCE_CODE:-$PWD}[0;m
[32;1m$ if ! docker info &>/dev/null; then # collapsed multi-line command[0;m
[32;1m$ function propagate_env_vars() { # collapsed multi-line command[0;m
[32;1m$ if [ -n "$CODECLIMATE_REGISTRY_USERNAME" ] && [ -n "$CODECLIMATE_REGISTRY_PASSWORD" ] && [ -n "$CODECLIMATE_PREFIX" ]; then # collapsed multi-line command[0;m
[32;1m$ docker pull --quiet "$CODE_QUALITY_IMAGE"[0;m
registry.gitlab.com/gitlab-org/ci-cd/codequality:0.96.0
[32;1m$ docker run --rm \ # collapsed multi-line command[0;m
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF: unknown.
section_end:1719397787:step_script
[0Ksection_start:1719397787:upload_artifacts_on_failure
[0K[0K[36;1mUploading artifacts for failed job[0;m[0;m
[32;1mUploading artifacts...[0;m
[0;33mWARNING: gl-code-quality-report.json: no matching files. Ensure that the artifact path is relative to the working directory (/builds/gl-demo-ultimate-khrechyshkina/tickets/zd543202)[0;m 
[31;1mERROR: No files to upload                         [0;m 
section_end:1719397788:upload_artifacts_on_failure
[0Ksection_start:1719397788:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
section_end:1719397789:cleanup_file_variables
[0K[31;1mERROR: Job failed: exit code 125
[0;m

It is not clear whether it is possible to achieve Codequality on rootless docker runner without sharing the host's docker socket inside the container, but there is a demand for it, see this internal customer ticket.