Make sure that turning a clipped group into a special group is handled properly

Currently it might break things, unclear. I think if we just destroy the clip membership everything should work.