Fix robots.txt
robots.txt
is generated by https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/robots.txt.erb.
With the way we deploy our versioned site, we end up with multiple robots.txt
:
- https://docs.gitlab.com/robots.txt
- https://docs.gitlab.com/14.7/robots.txt
- https://docs.gitlab.com/14.6/robots.txt
- https://docs.gitlab.com/14.5/robots.txt
- https://docs.gitlab.com/13.12/robots.txt
- https://docs.gitlab.com/12.10/robots.txt
However, only the one under the root domain matters: docs.gitlab.com/robots.txt
. So, it doesn't make any sense deploying it under the versions.
We want to exclude the versions from being indexed, and I see we had added this in !288 (diffs), but removed it in !1432 (diffs).
We should bring back !288 (diffs).