Skip to content

Tell search engines not to index pages/development sites

Steven McDonald requested to merge stevenmcdonald/fdroid-website:robots-txt-dev into master

The goal here is to add a robots.txt that disallows indexing of the development sites deployed to gitlab pages, hopefully I've gotten that right and the rule I changed isn't used for production.

  • add a robots.txt.noindex
  • copy that over as robots.txt for Gitlab pages deploys

I don't think this is a big deal for SEO, but it's a little embarrassing if development site start showing up in search results.

Merge request reports