Removing remote mirror remote fails because some refs aren't found

https://sentry.gitlab.net/gitlab/gitlabcom/issues/611586/

GRPC::Unknown: 2:Rugged::ReferenceError: reference 'refs/remotes/remote_mirror_1ba3692f326efe01b6011fee025e7218/doc-refactor-description-templates' not found
  grpc/generic/active_call.rb:31:in `check_status'
    fail GRPC::BadStatus.new_status_exception(
  grpc/generic/active_call.rb:181:in `attach_status_results_and_complete_call'
    recv_status_batch_result.check_status
  grpc/generic/active_call.rb:377:in `request_response'
    attach_status_results_and_complete_call(batch_result)
  grpc/generic/client_stub.rb:178:in `block in request_response'
    c.request_response(req, metadata: metadata)
  grpc/generic/interceptors.rb:170:in `intercept!'
    return yield if @interceptors.none?
...
(61 additional frame(s) were not displayed)

Gitlab::Git::CommandError: 2:Rugged::ReferenceError: reference 'refs/remotes/remote_mirror_1ba3692f326efe01b6011fee025e7218/doc-refactor-description-templates' not found
  gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'
    raise Gitlab::Git::CommandError.new(e)
  gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
    def wrapped_gitaly_errors(&block)
  gitlab/git/repository.rb:641:in `remove_remote'
    wrapped_gitaly_errors do
  repository.rb:503:in `method_missing'
    raw_repository.__send__(msg, *args, &block) # rubocop:disable GitlabSecurity/PublicSend
  gitlab/metrics/instrumentation.rb:161:in `block in method_missing'
    .measure { super }
...
(52 additional frame(s) were not displayed)

2:Rugged::ReferenceError: reference 'refs/remotes/remote_mirror_1ba3692f326efe01b6011fee025e7218/doc-refactor-description-templates' not found