Geo repository sync: Investigate out of sync and failures on GPRD site

We have about 2000 projects that are out of sync and 21,000+ that failed to sync.

We need to figure out why and whether these are real issues.

May be related to https://gitlab.com/gitlab-org/gitlab-ee/issues/4894.

Description	% repositories	Fixes
Not Found on disk	82%	Fixed by https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4677, https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4757
Repositories we just need to retry downloading again (empty reason, Error 500s, curl transfer error), but Geo fails to retry		Fixed by https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4727
Timed out		These might be repositories with lots of refs (one I tested appeared to have 6800+ and counting: a direct Gitaly fetch might help here). Upgrade to git 2.16: gitlab-org/omnibus-gitlab!2320 (merged), Don't use --prune on recovery: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4874
"could not read Username for TOKEN: terminal prompts disabled"		I suspect these fall in the same category of 2 or 3
Corrupted repositories/stale lock files		we'll need to somehow repair these or just rsync them