Restore fail from backup with missing content and 13:CreateRepositoryFromBundle: cmd wait failed: exit status 128

Summary

When restoring from a backup, some repositories are not restored with an error.

Backgroud

We periodically back up production gitlab and restore the copy to the test server.

Production server and test server are the same (differ only in name in /etc/hostname and url in /etc/gitlab/gitlab.rb)

Problem

When restoring from a successful backup, some repositories are not restored with errors

 * group1/repo1 ... Error: 13:CreateRepositoryFromBundle: cmd wait failed: fatal: pack has bad object at offset 10444426: inflate returned -5
error: index-pack died
fatal: bad object 98b7a1a56fff84ef99c99cb69a6415e278ca7b43
fatal: remote did not send all necessary objects
: exit status 128
[Failed] restoring hd/officeitdocs repository

 * group2/repo2 ... Error: 13:CreateRepositoryFromBundle: cmd wait failed: error: inflate: data stream error (incorrect data check)
fatal: pack has bad object at offset 232508335: inflate returned -3
error: index-pack died
fatal: bad object 5c11009c5e69770677d3a514707f658102e07151
fatal: remote did not send all necessary objects
: exit status 128
[Failed] restoring msi-scripts/nod32v6 repository

Оbservations

  1. If you re-restore from one backup, then the list of problematic repositories will repeat

  2. When restoring from different backups, the list of problematic repositories may change

  3. Problems can arise with both small and large repositories

What is the expected correct behavior?

Restore without errors and missing content

Software Versions

QEMU/KVM guest with 4 core and memory 16 Gb (production and test servers)

Self-hosting Omnibus GitLab CE 13.2.4

System information
System:          Ubuntu 18.04
Current User:    git
Using RVM:       no
Ruby Version:    2.6.6p146
Gem Version:     2.7.10
Bundler Version: 1.17.3
Rake Version:    12.3.3
Redis Version:   5.0.9
Git Version:     2.27.0
Sidekiq Version: 5.2.9
Go Version:      unknown

GitLab information
Version:         13.2.4
Revision:        136d3a02dca
Directory:       /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:      PostgreSQL
DB Version:      11.7
Using LDAP:      no
Using Omniauth:  yes
Omniauth Providers:

GitLab Shell
Version:          13.3.0
Repository storage paths:
- default:         /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git:               /opt/gitlab/embedded/bin/git