Prune robots.txt to remove unnecessary paths
After the move to the '-' scope is completed, robots.txt
will have some unnecessary lines that can be safely removed.
This is a follow-up issue that arose from the comments in this original issue.
After the move to the '-' scope is completed, robots.txt
will have some unnecessary lines that can be safely removed.
This is a follow-up issue that arose from the comments in this original issue.