Fix robots.txt parser

This is a parser conforming to https://www.robotstxt.org/orig.html#format and also supports https://datatracker.ietf.org/doc/html/draft-koster-rep#section-2.2 except the Allow: field.

Approver's checklist:

• The author has submitted the FSF Copyright Assignment and is listed in AUTHORS
• There is a test suite reasonably covering new functionality or modifications
• Function naming, parameters, return values, types, etc., are consistent with existing code
• This feature/change has adequate documentation added (if appropriate)
• No obvious mistakes / misspelling in the code