fatal: missing blob object error: remote did not send all necessary objects

Summary

Runner is unable to start job, reporting missing blob

Steps to reproduce

Not sure how to reproduce, happened to us recently on a bunch of projects, after upgrading both gitlab to 14.8.2 and runners to 14.7.0 (but can't be sure if that's the root cause)

Expected behavior

Job executes normally

Actual behavior

Output

Preparing environment
00:01
Running on runner-aaaaaa-project-1234-concurrent-0 via my-runner...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/group/project/.git/
fatal: missing blob object '1fc8d08b91012b7c1ea63f4494dc2ba41263f819'
error: remote did not send all necessary objects
Cleaning up project directory and file based variables

Environment description

Shared runner docker, or docker-in-docker

docker: Server Version: 20.10.12

Used GitLab Runner version

 gitlab-runner -version
Version:      14.7.0
Git revision: 98daeee0
Git branch:   14-7-stable
GO version:   go1.17.5
Built:        2022-01-19T17:11:48+0000
OS/Arch:      linux/amd64

also happened on runner 14.6.0

Possible fixes

the only way to fix this currently is to rm -rf /cache/* (<- or project specific folder below)

Button 'clear runner cache' doesn't work