Failed to request job: runner requestConcurrency meet builds=0

Summary

Runner cannot start jobs. It always states: Failed to request job: runner requestConcurrency meet builds=0

Steps to reproduce

Use the provided config.toml.

Register the runner and try to run a job

Example of a .gitlab-ci.yml that's not running

variables:
  NODE_ENV: 'test'

default:
  image: node:12.18.3-alpine
  interruptible: true
  cache:
    paths:
      - node_modules

stages:
  - prepare
  - test
  - build

install_dependencies:
  stage: prepare
  script:
    - apk add --no-cache --virtual .build-deps python make gcc g++ git
    - npm ci
  only:
    - merge_requests
  except:
    variables:
      - $CI_MERGE_REQUEST_TITLE =~ /^WIP:/

tests:
  stage: test
  services:
    - name: postgres:12.1-alpine
      alias: postgres
    - name: rabbitmq:3.8-rc-alpine
      alias: rabbitmq
  script:
    - npm test
  only:
    - merge_requests
  except:
    variables:
      - $CI_MERGE_REQUEST_TITLE =~ /^WIP:/

build_image:
  stage: build
  image: docker:19.03.11
  services:
    - name: docker:19.03.11-dind
  script:
    - docker run --rm -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY amazon/aws-cli ecr get-login-password --region $AWS_DEFAULT_REGION | docker login -u AWS --password-stdin 110340091796.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
    - DOCKER_BUILDKIT=1 docker build . --cache-from $IMAGE_REGISTRY --build-arg COMMIT=$CI_COMMIT_SHA -t $IMAGE_REGISTRY
    - docker push $IMAGE_REGISTRY
  only:
    - prod                                                                               

Actual behavior

With debug log_level: Failed to request job: runner requestConcurrency meet builds=0

With info log_level: Checking for jobs... failed status=502 Bad Gateway

This morning, before 12h15m it picked jobs just fine. This is the weirder part. I did nothing on the machine

Expected behavior

The jobs to run on my runner1 runner. Not on some Gitlab Ci shared runner, for example 'docker-auto-scale 0277ea0f'

Relevant logs and/or screenshots

job log

Feeding runners to channel                          builds=0
Failed to request job: runner requestConcurrency meet  builds=0 runner=...

Environment description

Custom gitlab runner on a local machine. Docker executor with version 19.03.12

config.toml contents

concurrent = 4
check_interval = 0
log_level = "debug"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "runner1"
  url = "https://gitlab.com/"
  token = "..."
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    privileged = true
    tls_verify = false
    image = "docker:19.03.11"
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0

Used GitLab Runner version

"Runtime platform                                  \u001b[0;m  arch\u001b[0;m=amd64 os\u001b[0;m=linux pid\u001b[0;m=6 revision\u001b[0;m=738bbe5a version\u001b[0;m=13.3.1\n","stream":"stderr","time":"2020-09-11T12:03:31.438927845Z"}
{"log":"Starting multi-runner from /etc/gitlab-runner/config.toml...\u001b[0;m  builds\u001b[0;m=0\n","stream":"stderr","time":"2020-09-11T12:03:31.438956531Z"}
{"log":"Running in system-mode.

Possible fixes

Did not work