Praefect crashes when attempting to replicate a repository that does not exist

Praefect crashes when a replication job is run on a repository path that does not exist.

I believe the path Praefect is attempting to replicate is a deleted project. It's not clear to me where the +22+deleted is coming from, unless we're appending that to the path of a project as it is being deleted.

{
  "from_storage": "praefect-gitaly-1",
  "level": "info",
  "msg": "processing replication job",
  "pid": 21099,
  "relative_path": "@hashed/78/5f/785f3ec7eb32f30b90cd0fcf3657d388b5ff4297f2f9716ff66e9b69c05ddd09+22+deleted.git",
  "replication_job_id": 1,
  "time": "2019-11-04T17:42:54Z",
  "to_storage": "praefect-gitaly-2"
}
{
  "error": "rpc error: code = NotFound desc = GetRepoPath: not a git repository '/var/opt/gitlab/git-data/repositories/@hashed/78/5f/785f3ec7eb32f30b90cd0fcf3657d388b5ff4297f2f9716ff66e9b69c05ddd09+22+deleted.git'",
  "level": "error",
  "msg": "error when replicating",
  "pid": 21099,
  "replication_job_id": 1,
  "time": "2019-11-04T17:42:54Z"
}
{
  "level": "fatal",
  "msg": "rpc error: code = NotFound desc = GetRepoPath: not a git repository '/var/opt/gitlab/git-data/repositories/@hashed/78/5f/785f3ec7eb32f30b90cd0fcf3657d388b5ff429
7f2f9716ff66e9b69c05ddd09+22+deleted.git'",
  "pid": 21099,
  "time": "2019-11-04T17:42:54Z"
}
{
  "level": "info",
  "msg": "Starting Praefect",
  "pid": 21623,
  "time": "2019-11-04T17:42:54Z",
  "version": "Praefect, version 1.70.0-31-g65913b93"
}

Full logs from Praefect and child Gitaly node attached below.

gitaly.log

praefect.log

/cc @pokstad1

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information