Push mirror can get stuck in "started" state while update job is no longer running

The push mirror from gitlab-ee on .com to dev was stuck and needed to manually failed in order to retry the mirror job. It appeared that job had been running for 24h.

To recover we needed to:

ee = Project.find_by_full_path("gitlab-org/gitlab-ee")
mirror = ee.remote_mirrors[0]
mirror.mark_as_failed("manually failed")

Stuck mirrors that require manual intervention

• 2019-06-13: Needed to unstick the mirror for gitlab-ee
• 2019-06-19: Needed to unstick the push mirrors for gitlab-ee and gitlab-ce
• 2019-06-25: Push mirrors were broken for 20 hours, needed to unstick gitlab-ee, gitlab-ce
• 2019-06-26: Push mirrors were broken for 12 hours, needed to unstick gitlab-ee
• 2019-06-27: Push mirrors were stuck for 15 hours, needed to unstick both gitlab-ee and gitlab-ce
• 2019-07-03: Push mirrors were stuck for 24 hours, needed to unstick both gitlab-ee and gitlab-ce
• 2019-07-05: Push mirrors were stuck for 22 hours, needed to unstick both gitlab-ee and gitlab-ce
• 2019-07-08: Push mirror stuck for 4 hours, needed to unstick gitlab-ee
• 2019-07-09: Push mirror stuck for 8 hours, needed to unstick gitlab-ee
• 2019-07-09: Push mirror stuck for 1 hour, needed to unstick gitlab-ee
• 2019-07-10: Push mirror stuck for 9 hours, needed to unstick gitlab-ee
• 2019-07-10: Push mirror stuck for 15 hours, needed to unstick gitlab-ee
• 2019-07-12: Push mirror stuck for 16 hours, needed to unstick gitlab-ee
• 2019-07-15: Push mirror stuck for 22 hours, needed to unstick gitlab-ee and gitlab-ce
• 2019-07-17: Push mirror stuck for 8 hours, needed to unstick gitlab-ee
• 2019-07-18: Push mirror stuck on gitlab-ee and gitlab-ee
• 2019-07-18: Push mirror stuck for the second time on gitlab-ee and gitlab-ee
• 2019-07-19: Push mirror stuck for 2 hours, needed to unstick gitlab-ce

/cc @reprazent