The source project of this merge request has been removed.
Update robots.txt disallow to allow users/groups starting with "s"
What does this MR do?
Update robots.txt "Disallow: /s" line to "Disallow: /s/"
Are there points in the code the reviewer needs to double check?
No
Why was this MR needed?
Currently any Project that is under a username or group name that starts with "s" cannot be indexed because of this disallow line.
For example on https://gitlab.com looking at the projects with the most stars there is this project: https://gitlab.com/somerobots/Trident
If you do a Google search for "site:gitlab.com Trident", the project comes up but with "A description for this result is not available because of this site's robots.txt"
Adding the trailing slash will still disallow all snippet short urls, but start allowing projects where the URI begins with "s"
Screenshots (if relevant)
N/A
Does this MR meet the acceptance criteria?
-
CHANGELOG entry added - [N/A] Documentation created/updated
- [N/A] API support added
- Tests
- [N/A] Added for this feature/bug
-
All builds are passing
-
Conform by the merge request performance guides -
Conform by the style guides -
Branch has no merge conflicts with master
(if you do - rebase it please) -
Squashed related commits together
What are the relevant issue numbers?
none