Villagers are not able to use ladders in the Norman villages.

This is still an issue in B7

Workaround for players- remove the ladders and they will teleport.