[SKIP RSPEC FAIL-FAST] Expand Robots.txt parser with Allow rules

What does this MR do?

In this MR we extend the Robots.txt parser by parsing Allow rules, also use the precedence order of those over Disallow ones, and add the functionality to have rules ending in $.

It also adds a spec to cover all existing cases.

Refs #36836 (closed)

Does this MR meet the acceptance criteria?

Conformity

• Code review guidelines
• Merge request performance guidelines
• Style guides
• Separation of EE specific content