Kaniko unable to build with gitlab-runner 11.11.0-rc2

Summary

When trying to use Kaniko to build Docker images from my project, the execution of "/kaniko/executor" hangs, with no output at all (even when adding the "--verbosity debug" commandline flag to Kaniko.

I tried multiple Kaniko image versions (debug, debug-v0.9.0) and the same thing happens. I tried Kaniko entrypoint as entrypoint: [""] and entrypoint: ["/busybox/sh", "-c"] and neither of them works, the command hangs in both cases.

Locally (on my laptop), using the same Dockerfile (and local Docker, not Kaniko) I can build the image.

When couple of weeks ago I pushed changes to the same repository, gitlab-runner was on version 11.10.0-rc2, and it worked flawlessly. I can't see any other difference between the runs.

Steps to reproduce

I use the following GitLab CI definition, build-java succeeds, build_docker_management hangs.

.gitlab-ci.yml

stages:
  - build
  - docker 

build_java:
  stage: build
  image: adoptopenjdk/openjdk11-openj9:jdk-11.0.3.7_openj9-0.14.0
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends libatomic1 && rm -rf /var/lib/apt/lists/*
    - cd java
    - export GRADLE_OPTS="-Dorg.gradle.daemon=false"
    - export GRADLE_USER_HOME=$PWD/.gradle
    - chmod +x ./gradlew
  script:
    - ./gradlew clean build codeCoverageReport
  artifacts:
    paths:
      - java/servers/server-management/build/libs/management-*.jar
      - java/servers/server-art/build/libs/art-*.jar
      - java/servers/server-links/build/libs/links-*.jar
  cache:
    key: "$CI_COMMIT_REF_NAME"
    paths:
      - java/.gradle

build_docker_management:
  stage: docker
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: ["/busybox/sh", "-c"]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - mkdir docker_staging
    - cp java/servers/server-management/build/libs/management-*.jar ./docker_staging/
    - cp java/servers/server-management/src/main/docker/Dockerfile ./docker_staging/
    - /kaniko/executor --verbosity debug --context $CI_PROJECT_DIR/docker_staging --dockerfile $CI_PROJECT_DIR/docker_staging/Dockerfile --destination $CI_REGISTRY_IMAGE/management:$CI_COMMIT_TAG --destination $CI_REGISTRY_IMAGE/management:DEV-$CI_PIPELINE_IID
  dependencies:
    - build_java

Actual behavior

Kaniko execution hangs without any output.

Expected behavior

Kaniko should build a Docker image and push it to GitLab Registry.

Environment description

I'm using shared runners.

Used GitLab Runner version

Running with gitlab-runner 11.11.0-rc2 (7f58b1ec)
  on docker-auto-scale fa6cab46
Using Docker executor with image gcr.io/kaniko-project/executor:debug ...
Pulling docker image gcr.io/kaniko-project/executor:debug ...
Using docker image sha256:bb44aed4398725f02a5d7ad1abd725a83c4a1a58e76a17cce7164cb2a01fe375 for gcr.io/kaniko-project/executor:debug ...
Running on runner-fa6cab46-project-11376427-concurrent-0 via runner-fa6cab46-srm-1557324870-a7233bd6...

Root Cause

For GitLab Runners on GCP we have /dummy-sys-class-dmi-id:/sys/class/dmi/id:ro as a volume mount so kaniko does not hang. With !1261 (merged) we introduced a regression #4250 (closed) which prevented this volume from being mounted.

Timeline

May 22nd - We identified the issue May 23rd - We rolled back Runner fleet to 11.10.1 May 29th - We updated Runner fleet to 11.11.1