Add note on link checker and robots.txt
What problem do you want to solve?
The current link checker does not respect robots.txt
.
What is the proposed solution?
Given the use case, which is internal testing of a site, robots.txt
should be ignored, but should add a note to the documentation to be clear.