Check if robots.txt parser complies with standardized format
At https://github.com/rockdaboot/wget2/issues/263 we came up with the ida of checking the existing parser against the robots.txt proposed standard.
Some people at Google created an RFC draft. I guess it won't change much until it is finalized, so we can create a new parser.
Edited by Tim Rühsen