Saving cache for failed job provokes "Missing /usr/bin/gitlab-runner-helper. Creating cache is disabled."

Summary

I tried the feature from 13.5.0 that allows specifying "cache:when: always" (related to gitlab!41822 (merged)).

However, I got the following issue:

Saving cache for failed job
Missing /usr/bin/gitlab-runner-helper. Creating cache is disabled.

Version used is Gitlab.com's CI, running with gitlab-runner 13.5.0-rc2 (71c90c86) on docker-auto-scale 0277ea0f.

You can see the failed job here: https://gitlab.com/kazarma/matrix_app_service.ex/-/jobs/814385282

And the corresponding gitlab-ci.yml here: https://gitlab.com/kazarma/matrix_app_service.ex/-/blob/024bd95129fc0c9a705aafe14032270c5ff06aa8/.gitlab-ci.yml

Please note the cache system works well otherwise as I had built the cache previously by using a non-failing stage in a previous version of the CI: https://gitlab.com/kazarma/matrix_app_service.ex/-/jobs/813328102 (with corresponding CI file: https://gitlab.com/kazarma/matrix_app_service.ex/-/blob/7d2a13cd79b83bfe8f65c008de56457f604816e3/.gitlab-ci.yml)
Also note this is probably not related to the DEBUG env variable issue since there's none. See debug CI with the export script to get all env vars. https://gitlab.com/kazarma/matrix_app_service.ex/-/jobs/814436572

Steps to reproduce

You can see the failed job here: https://gitlab.com/kazarma/matrix_app_service.ex/-/jobs/814385282

Problematic part is that one:

dialyzer:
  stage: dialyzer
  script:
    - mix dialyzer
  needs: [] # start dialyzing in parallel
  cache:
    when: always
    paths:
      - _build
      - deps
      - mix.lock
      - priv/plts/*.plt
  variables:
    MIX_ENV: dev

.gitlab-ci.yml

stages:
  - "test"
  - "format"
  - "credo"
  - "dialyzer"
  - "docs"

image: bitwalker/alpine-elixir:1.10.4

before_script:
  - mix deps.get
  - mix compile

test:
  stage: test
  script:
    - mix test --cover
  artifacts:
    when: always
    paths:
      - cover
    reports:
      junit: _build/test/lib/matrix_app_service/test-junit-report.xml
      cobertura: coverage.xml
  cache:
    key: test
    when: always
    paths:
      - _build
      - deps
      - mix.lock
  variables:
    MIX_ENV: test

format:
  stage: format
  script:
    - mix format --check-formatted
  needs:
    - "test"
  cache:
    policy: pull # cache should have already been built at earlier stage, speed up by not pushing cache at the end
    key: test
    paths:
      - _build
      - deps
      - mix.lock
  variables:
    MIX_ENV: test

credo:
  stage: credo
  script:
    - mix credo --strict | tee credo.log
  artifacts:
    when: always
    paths:
      - "credo.log"
  needs:
    - "test"
  cache:
    policy: pull 
    key: test
    paths:
      - _build
      - deps
      - mix.lock
  variables:
    MIX_ENV: test

dialyzer:
  stage: dialyzer
  script:
    - mix dialyzer
  needs: [] # start dialyzing in parallel
  cache:
    when: always
    paths:
      - _build
      - deps
      - mix.lock
      - priv/plts/*.plt
  variables:
    MIX_ENV: dev

pages:
  stage: docs
  script:
    - mix docs -o public
  cache:
    key: dev
    policy: pull
    paths:
      - _build
      - deps
      - mix.lock
      - priv/plts/*.plt
  needs:
    - "dialyzer"
  artifacts:
    paths:
      - public
  only:
    - master
  variables:
    MIX_ENV: dev

Actual behavior

Saving cache for failed job
Missing /usr/bin/gitlab-runner-helper. Creating cache is disabled.

Expected behavior

Saving cache for failed job

Then upload of such cache.

Relevant logs and/or screenshots

job log

Running with gitlab-runner 13.5.0-rc2 (71c90c86)
  on docker-auto-scale 0277ea0f
Resolving secrets
00:00
Preparing the "docker+machine" executor
00:14
Using Docker executor with image bitwalker/alpine-elixir:1.10.4 ...
Pulling docker image bitwalker/alpine-elixir:1.10.4 ...
Using docker image sha256:a14eda2625d12f6a7afb886d17b05ed5ccd6a732728d5b528b15a5954d18e1d6 for bitwalker/alpine-elixir:1.10.4 with digest bitwalker/alpine-elixir@sha256:bf447af6bf018632ca13ef3e52fa60c6d2728eb5d8969660de98d641ff570382 ...
Preparing environment
00:02
Running on runner-0277ea0f-project-18570605-concurrent-0 via runner-0277ea0f-srm-1603803996-bd952305...
Getting source from Git repository
00:02
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/kazarma/matrix_app_service.ex/.git/
Created fresh repository.
Checking out 024bd951 as 8-improve-ci...
Skipping Git submodules setup
Restoring cache
00:02
Checking cache for default...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/18570605/default 
Successfully extracted cache
Executing "step_script" stage of the job script
00:16
$ mix deps.get
Resolving Hex dependencies...
Dependency resolution completed:
Unchanged:
  bunt 0.2.0
  certifi 2.5.2
  cobertura_cover 0.9.0
  cowboy 2.7.0
  cowlib 2.8.0
  credo 1.4.1
  dialyxir 1.0.0
  earmark_parser 1.4.10
  erlex 0.2.6
  ex_doc 0.22.6
  hackney 1.16.0
  idna 6.0.1
  jason 1.2.0
  junit_formatter 3.1.0
  makeup 1.0.3
  makeup_elixir 0.14.1
  metrics 1.0.1
  mime 1.3.1
  mimerl 1.2.0
  nimble_parsec 0.6.0
  parse_trans 3.3.0
  phoenix 1.5.1
  phoenix_pubsub 2.0.0
  plug 1.10.0
  plug_cowboy 2.2.1 RETIRED!
    (invalid) Broken telemetry support
  plug_crypto 1.1.2
  poison 4.0.1
  polyjuice_client 0.2.3
  polyjuice_util 0.1.0
  ranch 1.7.1
  ssl_verify_fun 1.1.6
  telemetry 0.4.1
  telemetry_metrics 0.4.2
  telemetry_poller 0.5.0
  unicode_util_compat 0.5.0
A new Hex version is available (0.20.5 < 0.20.6), please update with `mix local.hex`
All dependencies are up to date
$ mix compile
Compiling 16 files (.ex)
warning: Polyjuice.Client.LowLevel.create/2 is undefined (module Polyjuice.Client.LowLevel is not available or is yet to be defined)
  lib/matrix_app_service/client.ex:36: MatrixAppService.Client.client/1
warning: MatrixAppServiceWeb.Endpoint.config_change/2 is undefined (module MatrixAppServiceWeb.Endpoint is not available or is yet to be defined)
  lib/matrix_app_service/application.ex:40: MatrixAppService.Application.config_change/3
warning: Polyjuice.Client.LowLevel.register/2 is undefined (module Polyjuice.Client.LowLevel is not available or is yet to be defined)
  lib/matrix_app_service/client.ex:105: MatrixAppService.Client.register/2
warning: Polyjuice.Client.Room.create_alias/3 is undefined or private
  lib/matrix_app_service/client.ex:63: MatrixAppService.Client.create_alias/3
warning: Polyjuice.Client.Room.create_room/2 is undefined or private
  lib/matrix_app_service/client.ex:49: MatrixAppService.Client.create_room/2
Generated matrix_app_service app
$ mix dialyzer
Finding suitable PLTs
Checking PLT...
[:asn1, :certifi, :compiler, :cowboy, :cowlib, :crypto, :eex, :elixir, :hackney, :idna, :jason, :kernel, :logger, :metrics, :mime, :mimerl, :parse_trans, :phoenix, :phoenix_pubsub, :plug, :plug_cowboy, :plug_crypto, :poison, :polyjuice_client, :polyjuice_util, :public_key, :ranch, :runtime_tools, :ssl, :ssl_verify_fun, :stdlib, :syntax_tools, :telemetry, :telemetry_metrics, :telemetry_poller, :unicode_util_compat]
Looking up modules in dialyzer.plt
Finding applications for dialyzer.plt
Finding modules for dialyzer.plt
Checking 1022 modules in dialyzer.plt
ignore_warnings: .dialyzer_ignore.exs
Starting Dialyzer
[
  check_plt: false,
  init_plt: '/builds/kazarma/matrix_app_service.ex/priv/plts/dialyzer.plt',
  files: ['/builds/kazarma/matrix_app_service.ex/_build/dev/lib/matrix_app_service/ebin/Elixir.MatrixAppService.Adapter.Room.beam',
   '/builds/kazarma/matrix_app_service.ex/_build/dev/lib/matrix_app_service/ebin/Elixir.MatrixAppService.Adapter.Transaction.beam',
   '/builds/kazarma/matrix_app_service.ex/_build/dev/lib/matrix_app_service/ebin/Elixir.MatrixAppService.Adapter.User.beam',
   '/builds/kazarma/matrix_app_service.ex/_build/dev/lib/matrix_app_service/ebin/Elixir.MatrixAppService.Application.beam',
   '/builds/kazarma/matrix_app_service.ex/_build/dev/lib/matrix_app_service/ebin/Elixir.MatrixAppService.Client.beam',
   ...],
  warnings: [:unknown]
]
Total errors: 9, Skipped: 0, Unnecessary Skips: 0
done in 0m2.55s
:0:unknown_function
Function MatrixAppServiceWeb.Endpoint.config_change/2 does not exist.
________________________________________________________________________________
:0:unknown_function
Function Mix.env/0 does not exist.
________________________________________________________________________________
:0:unknown_function
Function Polyjuice.Client.LowLevel.create/2 does not exist.
________________________________________________________________________________
:0:unknown_function
Function Polyjuice.Client.LowLevel.register/2 does not exist.
________________________________________________________________________________
:0:unknown_type
Unknown type: MatrixAppService.Client.LowLevel.create_opts/0.
________________________________________________________________________________
:0:unknown_type
Unknown type: Polyjuice.Client.LowLevel.register_opts/0.
________________________________________________________________________________
:0:unknown_type
Unknown type: Polyjuice.Client.LowLevel.t/0.
________________________________________________________________________________
lib/matrix_app_service/client.ex:49:call_to_missing
Call to missing or private function Polyjuice.Client.Room.create_room/2.
________________________________________________________________________________
lib/matrix_app_service/client.ex:63:call_to_missing
Call to missing or private function Polyjuice.Client.Room.create_alias/3.
________________________________________________________________________________
done (warnings were emitted)
Halting VM with exit status 2
Saving cache for failed job
00:01
Missing /usr/bin/gitlab-runner-helper. Creating cache is disabled.
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 1

Environment description

Using shared Runners on GitLab.com for the kazarma/matrix_app_service.ex project.

Used GitLab Runner version

Running with gitlab-runner 13.5.0-rc2 (71c90c86)
  on docker-auto-scale 0277ea0f
Resolving secrets
00:00
Preparing the "docker+machine" executor
00:14
Using Docker executor with image bitwalker/alpine-elixir:1.10.4 ...
Pulling docker image bitwalker/alpine-elixir:1.10.4 ...
Using docker image sha256:a14eda2625d12f6a7afb886d17b05ed5ccd6a732728d5b528b15a5954d18e1d6 for bitwalker/alpine-elixir:1.10.4 with digest bitwalker/alpine-elixir@sha256:bf447af6bf018632ca13ef3e52fa60c6d2728eb5d8969660de98d641ff570382 ...
Preparing environment
00:02
Running on runner-0277ea0f-project-18570605-concurrent-0 via runner-0277ea0f-srm-1603803996-bd952305...

Possible fixes

Maybe it's the runner version not recent enough? (gitlab-runner 13.5.0-rc2 (71c90c86)) Related code is in this MR supposedly: gitlab!41822 (merged)

Also note this is probably not related to the DEBUG env variable issue since there's none. See debug CI with the export script to get all env vars: https://gitlab.com/kazarma/matrix_app_service.ex/-/jobs/814436572

Thank you!

Edited Oct 27, 2020 by Hadrien