Skip to content

Block crawlers from indexing dev site

Sarah German requested to merge noindex-metatag into main

What does this MR do and why?

Our robots.txt file is not actually blocking crawlers.

This is because crawlers only look for these at example.com/robots.txt, not nested in a folder structure like the dev site URL.

To avoid pages being indexed, we'll need to include a metatag site-wide.

Slack thread: https://gitlab.slack.com/archives/CETG54GQ0/p1713216997994149

Merge request acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports