With more objects it gets exponentionally slower

I'm trying to move many of icons to separate layers. While the number of layers increase the extension gets slower and slower. It seems to loop through every layer at every object transfer. Can't find this in the code though.