git clone fails with "not our ref" to active-active gitlab-ee cluster

fatal: remote error: upload-pack: not our ref ffff000011112222333344445555666677778888
fatal: the remote end hung up unexpectedly

Summary

gitlab-ee active-active cluster receives a git push to an NFS backed git-data store. gitlab-runner picks up a build job for a very recently pushed commit (speculation) and upon trying to fetch it over http, gets requests split over the gitlab application server cluster at which point part of the cluster serves bad data to the runner and causes the job to fail.

Steps to reproduce

This happens on gitlab-runner using the docker executor, after the container is fetched, when the code repository is cloned. Happens with "Reinitialized existing Git repository" as well, happens intermittently, suspect happens when git clone is routed to multiple application servers with nfs backed git-data.

.gitlab-ci.yml

test runner:
	image: debian:stretch
	script:
		- echo hello world

Actual behavior

Upon encountering a git clone error "remote error: upload-pack: not our ref" the job fails and gives up, and requires human intervention to retry the job at which point it works (reliably AFAICT).

Expected behavior

Gitlab-runner makes a good faith effort to retry synchronization-based failures that occur outside of the tasks controlled by the repository .gitlab-ci-yml file. Or otherwise, automatically retrying a build job that fails over an internal error that is likely due to synchronization issues.

There are comments that this error is http specific due to the way the http git fetch machinery does multiple requests, where as the ssh clone uses a single connection. Possibly the expected behaviour is that a connection protocol who's reliability isn't tied to the topology of the gitlab application is used (like ssh or a modified http transport that uses a single connection?). I'm open to the gitlab-runner team's export input on how this should work.

Relevant logs and/or screenshots

job log

Running with gitlab-runner 11.10.1 (1f513601)
  on gitlab-runner-worker REDACTED
Using Docker executor with image mydockerregistry.example.org/name/namespace:latest ...
Pulling docker image mydockerregistry.example.org/name/namespace:latest ...
Using docker image sha256:0000111122223333444455556666777788889999aaaabbbbccccddddeeeeffff for mydockerregistry.example.org/name/namespace:latest ...
Running on runner-REDACTED-project-9999-concurrent-0 via runner-REDACTED-gitlab-runner-worker-1111222233-aabbccdd...
Reinitialized existing Git repository in /builds/mygitrepo/mygitrepo/.git/
Fetching changes...
fatal: remote error: upload-pack: not our ref aaaabbbbccccddddeeeeffff0000111122223333
fatal: the remote end hung up unexpectedly
ERROR: Job failed: exit code 1

Environment description

custom installation (runner bastion with an ephemeral single-use gitlab runner behind it) docker executor docker is provided by a managed provider who does not expose version information or runner configuration.

config.toml contents

<unavailable>

Used GitLab Runner version

Running with gitlab-runner 11.10.1 (1f513601)
  on gitlab-runner-worker REDACTED
Using Docker executor with image REDACTED

-->

Possible fixes

Comments from forums suggest there isn't a simple fix in the git protocol, I suspect gitlab might be able to assert an application node has the tip update, or otherwise introduce a delay and try to flush the git ref cache to try to serve a request that should be good, idk. https://github.com/git/git/blob/aa25c82427ae70aebf3b8f970f2afd54e9a2a8c6/upload-pack.c#L615