Remove refs we don't care during housekeeping or after importing or mirroring
As far as I could tell, the only refs we really need are:
The rest are not needed, and we should be able to get rid of them. They could be coming from importing or mirroring. We might need those refs to be available to during importing, but certainly not needed after importing is done.