Skip to content

Document correct way to add custom robots.txt

Matthias Baur requested to merge syseleven/omnibus-gitlab:robots.txt into master

What does this MR do?

Fix the documentation of how to add a custom robots.txt

The old version just didn't work:

$ export GITLAB_HOME=$PWD
$ sudo docker run --detach \
  --hostname gitlab.example.com \
  --publish 443:443 --publish 80:80 \
  --name gitlab \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ee:latest
...
$ curl -H 'Host: gitlab.example.com' localhost/robots.txt
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /

# Add a 1 second delay between successive requests to the same server, limits resources used by crawler
# Only some crawlers respect this setting, e.g. Googlebot does not
# Crawl-delay: 1

# Based on details in https://gitlab.com/gitlab-org/gitlab/blob/master/config/routes.rb,
# https://gitlab.com/gitlab-org/gitlab/blob/master/spec/routing, and using application
...
$ echo "nginx['custom_gitlab_server_config'] = 'rewrite ^/robots.txt /var/opt/gitlab/robots.txt last;'" | sudo tee -a config/gitlab.rb
nginx['custom_gitlab_server_config'] = 'rewrite ^/robots.txt /var/opt/gitlab/robots.txt last;'
$ docker restart gitlab
gitlab
$ curl -H 'Host: gitlab.example.com' localhost/robots.txt
<html><body>You are being <a href="http://gitlab.example.com/users/sign_in">redirected</a>.</body></html>

Related issues

closes #3955 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
  • trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Tests added
  • Integration tests added to GitLab QA
  • Equivalent MR/issue for the GitLab Chart opened
Edited by Achilleas Pipinellis

Merge request reports