On startup, Praefect can't talk to gitaly nodes before it picks up replication jobs

During our demo on 3/20/20, we observed that when praefect is brought up when the sql replication queue has jobs in ready, praefect picks up the jobs immediately and tries to replicate, but failed because it couldn't talk to gitaly.

To reproduce:

  1. bring praefect down
  2. create replication jobs on the sql database
  3. bring praefect up

in the logs, praefect will complain that it cannot talk to gitaly, and the jobs will fail.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information