Some projects stuck in pending_delete
### Summary
Some projects are partially deleted and stuck in `pending_delete` state... reopening of https://gitlab.com/gitlab-org/gitlab-ce/issues/20984
### Steps to reproduce
Unknown
### What is the current *bug* behavior?
We have `Project.unscoped.where(pending_delete: true).map(&:name_with_namespace).count` returning 14 for us. In one case we have a project that a user deleted but it's still showing in their group. Trying to load the project returns 404. Doesn't show at all in admin section for projects and on disk it has been renamed `projectname+00+deleted.git` where `00` is it's ID.
### What is the expected *correct* behavior?
Delete from UI and disk
### Possible fixes
Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/20984, https://gitlab.com/gitlab-org/gitlab-ce/issues/27120, https://gitlab.com/gitlab-org/gitlab-ce/issues/23477, https://gitlab.com/gitlab-org/gitlab-ce/issues/17893
### Project stuck in deletion
* [Kibana link](https://log.gprd.gitlab.net/app/r?l=DISCOVER_APP_LOCATOR&v=8.12.1&lz=N4IgjgrgpgTgniAXKSsGJCANCANgQwDsBzCfYqJEAa2nhAF8cBnAexgBckBtbkAKzaEAdBwCWAW0o4AJlGYBjEAF1lOBa1wQJhZjwFDhUAB4KoAB3GsRU5s3LSD1o6YtWRCgnexORJs5ZizgBG%2BArUHDBhjoLOUhz4wuYwrPxQClw4sTZQCcIK%2BLi4sAD6YjIqOGKEcsZUAIIA6gBy9QBCACIASgDuzR0QAIoAEgAqEgBMHMMAbLg%2B1RywAG6FVPgQHKw%2BAGZiuEsweojcoPH4SKAyYvbBxRWIO4XMUDiEUMT4S0hPuC84hTE%2BGOhAgRSqNRMDRa7W6fQGI3GU1m8xw1Cg6F8LgC7nyXj0OD2UFwDyx%2FjcQQ8%2BJ8qy0lAwJhuHAJIA4cHM9JAjOYzMYOFQ9GQXOMTOOoCJJKo2WxFOcnmBegYTBAABIeV96aAeexOfhzOYAMoJb5KrBnXIXIXVWrQ1qdXr9IZjSbTOY%2BNkcqjmAAWURePnRmOl53yhWKMDKMmEgZ67AqOHM%2BCiEjF4DomIACik0hkOvJIqw4I12OiYHyQNdbvcfs9XiB3p9vo9awDcECQWDcMqBegzl8FN6Sj6%2FZqsSGCkVSuVoxjYzBSVnUukOHmeSkiyXYIxlWrjaPtTBdfqjRrt2aQOdLiBrVCME07XDHYiXSj3ezOdzeWiMVLDOTAnK%2BIznAc7xhWNz4HcUAPL8%2Fz1h8p7Nn8daAsCSCguCIC0tAVCfoq%2FLpleeFXhKpLSv%2BuLynYwGgWeqrqk2WpbIe6zHnu25qCA3rlFAADC3pJlwSFwcEh74NQMisD0hAAGJiMSZF%2Fq4AFUgqNFxj4MhfPgABq8k9AAkqS96wg6CLOsibo4OIUhdEQFAkSkEhUIQUkALQAIxgVsLlSeWh47IezDegZhCHLSV7YfSMwAAxxTFCYbP6iCRNASpAA%3D%3D%3D)
* [Sentry link](https://new-sentry.gitlab.net/organizations/gitlab/issues/?project=3&query=is%3Aunresolved+transaction%3ASidekiq%2FProjectDestroyWorker&referrer=issue-list&sort=freq&statsPeriod=14d)
/cc @stanhu @zj
issue