Commit 7c331449 authored by Deimos's avatar Deimos

robots.txt: Block /login from all scrapers

Various scrapers hit this a lot because it's on all pages with a
differing query variable (for the "from url" to redirect back to after
logging in).
parent bc527b0c
......@@ -4,6 +4,12 @@
# Scrapers from SEO/marketing-type services will be blocked. Tildes data is not a
# resource to be mined and sold.
# urls blocked for all scrapers
User-agent: *
Disallow: /login
# Specific, fully-blocked scrapers below
# https://ahrefs.com/robot
User-agent: AhrefsBot
Disallow: /
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment