Mysterious/unhandled Gitaly error when creating project from template

Summary

A 250-seat premium customer reported that (Zendesk, internal only) when creating a new project from a template, they see an empty repository when there should be some files. The project settings and configurations translate properly. Then, after a few minutes, the user refreshes and now sees the expected files.

I've looked through the logs provided and there is a strange Gitaly error (see below) which may or may not be the cause of the issue.

My best guess/theory is that this error that causes the project to be kicked out of the "importing" state and thus starts to display the repository contents. Meanwhile, Gitaly continues to create the repository in the background and eventually successfully does so. When the user refreshes, they are then able to see the expected files.

Steps to reproduce

Unknown.

Example Project

N/A

What is the current bug behavior?

Project created from template apparently completes import, but repository is empty. After awhile, repository files become available.

What is the expected correct behavior?

Project created from template completes import and repository files are available immediately.

Relevant logs and/or screenshots

The Gitaly error follows (I have obfuscated repository names/paths and converted \n to line breaks to facilitate readability):

2019-07-03_19:31:41.87643 gitpr01cala gitaly: time="2019-07-03T19:31:41Z" level=error msg="

From /gitlab-efs/gitlab-data/git-data/repositories/+gitaly/tmp/repo159703980/repo.bundle
 * [new ref]         refs/merge-requests/9/head  -> refs/merge-requests/9/head
 * [new ref]         refs/merge-requests/8/head  -> refs/merge-requests/8/head
 * [new ref]         refs/merge-requests/7/head  -> refs/merge-requests/7/head
 * [new ref]         refs/merge-requests/6/head  -> refs/merge-requests/6/head
 * [new ref]         refs/merge-requests/5/head  -> refs/merge-requests/5/head
 * [new ref]         refs/merge-requests/4/head  -> refs/merge-requests/4/head
 * [new ref]         refs/merge-requests/3/head  -> refs/merge-requests/3/head
 * [new ref]         refs/merge-requests/10/head -> refs/merge-requests/10/head
 * [new ref]         refs/keep-around/f70533203b150dde74a744be5e8f8f1c85665455 -> refs/keep-around/f70533203b150dde74a744be5e8f8f1c85665455
 * [new ref]         refs/keep-around/f65e70a0d77115447432e58f08742bd6cd37fd8b -> refs/keep-around/f65e70a0d77115447432e58f08742bd6cd37fd8b
 * [new ref]         refs/keep-around/f5e3c0edfa67ae7d041ec8f93d36f4ea058cdcd7 -> refs/keep-around/f5e3c0edfa67ae7d041ec8f93d36f4ea058cdcd7
 * [new ref]         refs/keep-around/e78befa64e2cc7f4318483ba92e5dec022863358 -> refs/keep-around/e78befa64e2cc7f4318483ba92e5dec022863358
 * [new ref]         refs/keep-around/df1533111d6e4743280114dd1cee8d827cf219bd -> refs/keep-around/df1533111d6e4743280114dd1cee8d827cf219bd
 * [new ref]         refs/keep-around/c8fffd743a2bf880cfa259b26f111634f430f0e2 -> refs/keep-around/c8fffd743a2bf880cfa259b26f111634f430f0e2
 * [new ref]         refs/keep-around/bcdf2a9b56c6e5856e2158d0d16645628b3b94cd -> refs/keep-around/bcdf2a9b56c6e5856e2158d0d16645628b3b94cd
 * [new ref]         refs/keep-around/b3af335be223e6d79e6fb727be633f32c19a9df6 -> refs/keep-around/b3af335be223e6d79e6fb727be633f32c19a9df6
 * [new ref]         refs/keep-around/a1f596a1b243ef147110722157641378887493b4 -> refs/keep-around/a1f596a1b243ef147110722157641378887493b4
 * [new ref]         refs/keep-around/979646f8d6b729ad57cf2f5199a36b2eaed507ae -> refs/keep-around/979646f8d6b729ad57cf2f5199a36b2eaed507ae
 * [new ref]         refs/keep-around/8ca608edcfbdba077cd1fc7536a95c2bdcd699d2 -> refs/keep-around/8ca608edcfbdba077cd1fc7536a95c2bdcd699d2
 * [new ref]         refs/keep-around/8bdeaf1c38cf8f36a0569a46efb479c653aa5405 -> refs/keep-around/8bdeaf1c38cf8f36a0569a46efb479c653aa5405
 * [new ref]         refs/keep-around/88be31c833afa15b5c37391ee471069a1b7c1ed4 -> refs/keep-around/88be31c833afa15b5c37391ee471069a1b7c1ed4
 * [new ref]         refs/keep-around/8668131552844992eda68643c657218f51f9ceed -> refs/keep-around/8668131552844992eda68643c657218f51f9ceed
 * [new ref]         refs/keep-around/7cbe4335b0cbae2bb28a2b9898acd7eb7707f5b3 -> refs/keep-around/7cbe4335b0cbae2bb28a2b9898acd7eb7707f5b3
 * [new ref]         refs/keep-around/7741460e4306b7234f2dc7dbd9cdbc0d4c84038e -> refs/keep-around/7741460e4306b7234f2dc7dbd9cdbc0d4c84038e
 * [new ref]         refs/keep-around/7288d1d5d2371d91192100991bd8255e04ebc205 -> refs/keep-around/7288d1d5d2371d91192100991bd8255e04ebc205
 * [new ref]         refs/keep-around/6984a5505278bd13e7133a546863841694fe9ad8 -> refs/keep-around/6984a5505278bd13e7133a546863841694fe9ad8
 * [new ref]         refs/keep-around/6948d191e7a7786ff2479a9a1a59c5acc556d75b -> refs/keep-around/6948d191e7a7786ff2479a9a1a59c5acc556d75b
 * [new ref]         refs/keep-around/67f7ff81ef05a7f9ccef7638ae62eb02e2bad206 -> refs/keep-around/67f7ff81ef05a7f9ccef7638ae62eb02e2bad206
 * [new ref]         refs/keep-around/63e667153ca50e5f43ea201d330a47eddfe67693 -> refs/keep-around/63e667153ca50e5f43ea201d330a47eddfe67693
 * [new ref]         refs/keep-around/5d4a002671a81faff6bfd6ba310a10e8a9fe2a1f -> refs/keep-around/5d4a002671a81faff6bfd6ba310a10e8a9fe2a1f
 * [new ref]         refs/keep-around/54fb9d8be80223ff40a179b4de815fc317e7663a -> refs/keep-around/54fb9d8be80223ff40a179b4de815fc317e7663a
 * [new ref]         refs/keep-around/4e33a12c80fc7a9e84bc509eac55227a627e7338 -> refs/keep-around/4e33a12c80fc7a9e84bc509eac55227a627e7338
 * [new ref]         refs/keep-around/4ba3bbe017d42dcfdc6acf8dd19bb05597110547 -> refs/keep-around/4ba3bbe017d42dcfdc6acf8dd19bb05597110547
 * [new ref]         refs/keep-around/4a32843c16c05c07cfcdc0f0aa2617f5f88ff2f6 -> refs/keep-around/4a32843c16c05c07cfcdc0f0aa2617f5f88ff2f6
 * [new ref]         refs/keep-around/453b7950773a2a8f9afb498a4211100a8422a2cc -> refs/keep-around/453b7950773a2a8f9afb498a4211100a8422a2cc
 * [new ref]         refs/keep-around/368c21cd4d4f518ecb642997dd33477de18b6d6c -> refs/keep-around/368c21cd4d4f518ecb642997dd33477de18b6d6c
 * [new ref]         refs/keep-around/34ccc4bcd0b4e720cd77268894c6f78ceebe9b89 -> refs/keep-around/34ccc4bcd0b4e720cd77268894c6f78ceebe9b89
 * [new ref]         refs/keep-around/297e2e61261f0d56561b3ce7b6d5382986c6d621 -> refs/keep-around/297e2e61261f0d56561b3ce7b6d5382986c6d621
 * [new ref]         refs/keep-around/0a06decb0075d9f6d45d56c61e2e841191853735 -> refs/keep-around/0a06decb0075d9f6d45d56c61e2e841191853735
 * [new ref]         refs/keep-around/01078f5a033a4c2d490767c3ed952d052b15c71e -> refs/keep-around/01078f5a033a4c2d490767c3ed952d052b15c71e
" 

correlation_id=Jf4rlz8eRQ5 grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq grpc.method=CreateRepositoryFromBundle grpc.request.fullMethod=/gitaly.RepositoryService/CreateRepositoryFromBundle grpc.request.glProjectPath=path/to/repository grpc.request.glRepository=project-344 grpc.request.repoPath=path/to/repository.git grpc.request.repoStorage=default grpc.request.topLevelGroup=path grpc.service=gitaly.RepositoryService grpc.start_time="2019-07-03T19:31:36Z" peer.address=@ span.kind=server system=grpc

Full logs from different instances of the problem are available in the Zendesk ticket link.

I have asked the customer to attempt running the underlying Git command manually when the problem occurs to get more diagnostic information, but I'm not sure if that is possible.

Output of checks

Customer reports that this happens on 11.11.2-ee and 12.0.3-ee.

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)