Block crawlers from indexing dev site
What does this MR do and why?
Our robots.txt file is not actually blocking crawlers.
This is because crawlers only look for these at example.com/robots.txt
, not nested in a folder structure like the dev site URL.
To avoid pages being indexed, we'll need to include a metatag site-wide.
Slack thread: https://gitlab.slack.com/archives/CETG54GQ0/p1713216997994149
Merge request acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this merge request.