Remove unwanted refs after importing a project

What does this MR do?

Remove unwanted refs after importing a project

Are there points in the code the reviewer needs to double check?

Should we also do this as regular housekeeping?

Why was this MR needed?

One of the goal of https://gitlab.com/gitlab-org/gitlab-ce/issues/36803

Does this MR meet the acceptance criteria?

• Changelog entry added, if necessary
• Documentation created/updated
• Tests added for this feature/bug
• Review
  • Has been reviewed by Backend

What are the relevant issue numbers?

Closes #36807 (closed)