Geo project/wiki sync shows no errors when Gitaly is down

Summary

During a Premium customer's Geo failover scenario (ZD - internal only), we noticed that after re-adding the initial node as a secondary, projects and wikis did not get replicated (and were 100% queued with no attempts). Manually trying to sync them showed a gitaly error, and once we fixed that, all projects got replicated, but there was no error and one could wait indefinitely for them to sync.

Steps to reproduce

Stop gitaly on the Geo secondary, create a new project on the primary - notice the project sync never gets scheduled and UI says "queued".

What is the current bug behavior?

Repositories don't get any sync attempt. UI also confusingly says "Queued" - when that's not the case (and can lead to investigating queuing problems - it did for us initially)

What is the expected correct behavior?

Repositories show as failed maybe, or some kind of error in the geo logs.

Possible fixes

I believe this happens because in logcursor, we only enqueue the job if the shard is healthy.