Gitlab Runner on Docker with Shell executor fails — Permission denied

Summary

Brand new Gitlab CE 13.9.1 on a clean Ubuntu Server 20.04.2.0. Using same procedures I was using on Gitlab CE 12. Set gitlab-runner in a docker container using shell. When running the job, returns error:

$ apt-get update -qq
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
ERROR: Job failed: exit status 1

Steps to reproduce

Pull last image:

docker pull gitlab/gitlab-runner:latest

Start GitLab Runner container mounting on local volume:

docker run -d \
--name gitlab-runner \
--restart always \
-v /srv/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest

Register runner, picking shell as executor:

docker run --rm -t -i \
-v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

Set pipeline on .gitlab-ci.yml:

.gitlab-ci.yml 
image: node:latest

before_script:
  - apt-get update -qq

stages:
  - install

install:
  stage: install
  script:
    - npm install --verbose

Actual behavior

$ apt-get update -qq
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
ERROR: Job failed: exit status 1

Expected behavior

Run the job

Relevant logs and/or screenshots

job log 
Add the job log

Environment description

config.toml contents 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "RUNNER_SHELL"
  url = REPLACED_URL
  token = REPLACED_RUNNER_TOKEN
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

Used GitLab Runner version

Possible fixes

Start GitLab Runner container mounting on Docker volume

Create volume

docker volume create gitlab-runner-config

Start GitLab Runner container

docker run -d \
--name gitlab-runner \
--restart always \
-v gitlab-runner-config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest

Register runner (picking shell again as executor)

docker run \
--rm -t -i \
-v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner register

Same results.

$ apt-get update -qq
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
ERROR: Job failed: exit status 1

Granting permissions to gitlab-runner

As per [https://stackoverflow.com/questions/50678061/in-gitlab-ci-the-gitlab-runner-choose-wrong-executor][2] and [https://docs.gitlab.com/runner/executors/shell.html#running-as-unprivileged-user][3], tried these solutions:

  1. move to docker
  2. grant user gitlab-runner the permissions he needs to run specified commands. gitlab-runner may run apt-get without sudo, also he will need perms for npm install and npm run.
  3. grant sudo nopasswd to user gitlab-runner. Add gitlab-runner ALL=(ALL) NOPASSWD: ALL (or similar) to /etc/sudoers on the machine gitlab-runner is installed and change the lines apt-get update to sudo apt-get update, which will execute them as privileged user (root).
  1. I need to use shell.
  2. I already did that with sudo usermod -aG docker gitlab-runner
  3. Tried as well with sudo nano /etc/sudoers, adding gitlab-runner ALL=(ALL) NOPASSWD: ALL, and using sudo apt-get update -qq in the pipeline, which results in bash: line 106: sudo: command not found

Also posted on https://stackoverflow.com/questions/66398460/gitlab-runner-with-docker-and-shell-error-permission-denied

Edited by Emile Collin