Prevent hard-failing a mirror or sending an access request from failing when the project has invited masters/owners

Issue introduced by https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4699