Speed up removing items from SPCanvasGroup
@marcjeanmougin as discussed on IRC yesterday.
Works around the performance issue from https://bugs.launchpad.net/inkscape/+bug/1652100 and https://bugs.launchpad.net/inkscape/+bug/1745763.
If you have spare time at you hands and want to dig into the code deeper in order to optimize it more fundamentally that's even better (in which case we probably won't need this). Otherwise it's ready to merge.