Speed up Group security access specs

This is the Group equivalent of 13ad9a74

See https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7779

Part of https://gitlab.com/gitlab-org/gitlab-ce/issues/24899