Provide Robots exclusion standard
Created by: andrechalom
We should provide a "robots.txt" to prevent Google / Yahoo / Bing / Baidu / etc bots from crawling the entire server
Created by: andrechalom
We should provide a "robots.txt" to prevent Google / Yahoo / Bing / Baidu / etc bots from crawling the entire server