Skip set_repo_refs to speed up the tests

What does this MR do?

Any created repository in tests sets repo refs, but it's not necessary. What about setting the refs only for the tests that actually need it?

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32556