Git pull hangs occasionaly

We have Gitlab Omnibus installation 13.8.4 (9fb9cbf50c3) and gitlab-runner helm chart (0.25.0 and 0.26.0-rc2) are tested. K8s 1.19.5

Standard gitlab-helper image and modified:

FROM gitlab/gitlab-runner-helper:x86_64-v13.9.0-rc2

RUN git config --global http.postBuffer 524288000
RUN git config --global core.compression 0

are tested.

git depth tested 0 (no shallow clone), 5, 50. git clone vs git fetch parameters are also tested.

gitlab-helper git pull over https hang indefinitely between sideband< PACK and sideband< 0000:

00:25:12.588801 pkt-line.c:80           packet:        fetch< shallow-info
00:25:12.588824 pkt-line.c:80           packet:        fetch< shallow c1b298f268af6ea982e144573a0ed24e6d90e640
00:25:12.588841 pkt-line.c:80           packet:        fetch< shallow 4f2f1545a0d068b0eb4ad00c11d3e29925ce4017
00:25:12.588853 pkt-line.c:80           packet:        fetch< shallow 820d3454357b7546b30b5446a92c1ae6ab74cd6f
00:25:12.588869 pkt-line.c:80           packet:        fetch< shallow 0a4ab05d7075be5927981830ec1ce60e049db730
00:25:12.588882 pkt-line.c:80           packet:        fetch< shallow 71335c116c76afef50b7fedd5e1738bf251c7fc9
00:25:12.588896 pkt-line.c:80           packet:        fetch< shallow 6a79f39cef1808806a257e57277b72cce8405c70
00:25:12.588909 pkt-line.c:80           packet:        fetch< shallow 26eb61bdbc784e42a443f1aa9fc9e9d76093603d
00:25:12.588922 pkt-line.c:80           packet:        fetch< shallow 9df1cb8d6f9d398ca9557f38d3a2ce7e9f83326e
00:25:12.588934 pkt-line.c:80           packet:        fetch< shallow 0183e3634e4923846caf461c87bf7f7e5835c61e
00:25:12.588946 pkt-line.c:80           packet:        fetch< shallow 3d15850079748da1141aa39592cdad0cf0d7e22c
00:25:12.588959 pkt-line.c:80           packet:        fetch< shallow 1c7abeb9764ff6f28c722c283dadd61b0f0a1248
00:25:12.588970 pkt-line.c:80           packet:        fetch< shallow 975ae5796649f404867bc22dda1bb15c3022f1f6
00:25:12.589013 pkt-line.c:80           packet:        fetch< shallow 22ca2794d91975d1ea6671b2fb302e0de786f371
00:25:12.589026 pkt-line.c:80           packet:        fetch< shallow f5b9d2814b5268e334e05210d54571abb30c40a8
00:25:12.589038 pkt-line.c:80           packet:        fetch< shallow 3ff4a41b083717070d76177014618c63bc3bc699
00:25:12.589049 pkt-line.c:80           packet:        fetch< 0001
00:25:12.589239 pkt-line.c:80           packet:        fetch< packfile
00:25:12.589970 pkt-line.c:80           packet:     sideband< PACK ...
00:25:12.590087 run-command.c:663       trace: run_command: git --shallow-file /builds/xxx/xxx/.git/shallow.lock index-pack --stdin --fix-thin '--keep=fetch-pack 36 on runner-sgsirl-6-project-208-concurrent-0l69lh' --pack_header=2,14416
00:25:12.591775 git.c:439               trace: built-in: git index-pack --stdin --fix-thin '--keep=fetch-pack 36 on runner-sgsirl-6-project-208-concurrent-0l69lh' --pack_header=2,14416
* Connection #0 to host gitlab.xxx.xxx left intact
00:30:14.103988 pkt-line.c:80           packet:     sideband< 0000
00:30:14.288238 run-command.c:663       trace: run_command: git --shallow-file /builds/xxx/xxx/.git/shallow.lock rev-list --objects --stdin --quiet --alternate-refs
00:30:14.289768 git.c:439               trace: built-in: git rev-list --objects --stdin --quiet --alternate-refs

Worst case scenario:

18 error: RPC failed; curl 56 OpenSSL SSL_read: Connection reset by peer, errno 104
19 fatal: the remote end hung up unexpectedly
20 fatal: early EOF
21 fatal: index-pack failed

P.S. About 10% of jobs of one repo affected (~50 jobs for pipeline). Kubernetes workers running payloads are completely random and no correlation runner -> node is observed