Directories with no read access are ignored by the default
Python mechanisms, do the same.

This may lead to inconsistencies between iter_modules and
find_modules, but this seems to match what the default
importer is doing in various calls when dealing with
unreadable files