Document correct way to add custom robots.txt
What does this MR do?
Fix the documentation of how to add a custom robots.txt
The old version just didn't work:
$ export GITLAB_HOME=$PWD
$ sudo docker run --detach \
--hostname gitlab.example.com \
--publish 443:443 --publish 80:80 \
--name gitlab \
--restart always \
--volume $GITLAB_HOME/config:/etc/gitlab \
--volume $GITLAB_HOME/logs:/var/log/gitlab \
--volume $GITLAB_HOME/data:/var/opt/gitlab \
--shm-size 256m \
gitlab/gitlab-ee:latest
...
$ curl -H 'Host: gitlab.example.com' localhost/robots.txt
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /
# Add a 1 second delay between successive requests to the same server, limits resources used by crawler
# Only some crawlers respect this setting, e.g. Googlebot does not
# Crawl-delay: 1
# Based on details in https://gitlab.com/gitlab-org/gitlab/blob/master/config/routes.rb,
# https://gitlab.com/gitlab-org/gitlab/blob/master/spec/routing, and using application
...
$ echo "nginx['custom_gitlab_server_config'] = 'rewrite ^/robots.txt /var/opt/gitlab/robots.txt last;'" | sudo tee -a config/gitlab.rb
nginx['custom_gitlab_server_config'] = 'rewrite ^/robots.txt /var/opt/gitlab/robots.txt last;'
$ docker restart gitlab
gitlab
$ curl -H 'Host: gitlab.example.com' localhost/robots.txt
<html><body>You are being <a href="http://gitlab.example.com/users/sign_in">redirected</a>.</body></html>
Related issues
closes #3955 (closed)
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion
Required
-
Merge Request Title, and Description are up to date, accurate, and descriptive -
MR targeting the appropriate branch -
MR has a green pipeline on GitLab.com -
Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks -
trigger-package
has a green pipeline running against latest commit
Expected (please provide an explanation if not completing)
-
Test plan indicating conditions for success has been posted and passes -
Documentation created/updated -
Tests added -
Integration tests added to GitLab QA -
Equivalent MR/issue for the GitLab Chart opened
Edited by Achilleas Pipinellis