Some projects stuck in pending_delete

Summary

Some projects are partially deleted and stuck in pending_delete state... reopening of https://gitlab.com/gitlab-org/gitlab-ce/issues/20984

Steps to reproduce

Unknown

What is the current bug behavior?

We have Project.unscoped.where(pending_delete: true).map(&:name_with_namespace).count returning 14 for us. In one case we have a project that a user deleted but it's still showing in their group. Trying to load the project returns 404. Doesn't show at all in admin section for projects and on disk it has been renamed projectname+00+deleted.git where 00 is it's ID.

What is the expected correct behavior?

Delete from UI and disk

Possible fixes

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/20984, https://gitlab.com/gitlab-org/gitlab-ce/issues/27120, https://gitlab.com/gitlab-org/gitlab-ce/issues/23477, https://gitlab.com/gitlab-org/gitlab-ce/issues/17893

/cc @stanhu @zj