Skip to content

Add Twitter and Facebook Crawlers

I use these two UA strings often to avoid cookie walls. Googlebot is often checked for valid PTR DNS records (cloudflare for example checks it very stringently), but those two often aren't. I guess the verification steps are not simple enough or not prominently documented enough for webdevs to implement (Twitter lists a range of IPs to whitelist, which is not very easy to find, and facebook tells you the ASN they come from. Both tell you not to hardcode IPs, as they change "often").

Merge request reports