-
Slack authored
It would appear that Google explicitely blocks requests with a Browser user agent if they originate from a server IP address (Google knows about). To circumvent this we would either need to use a proxy or a different user agent. The curl user agent seems to be working for now so let's try and use that. Additionally it seems that Google returns slightly altered content depending on the originating IP address of the request. This might make reliably unit tests impractical. If it turns out that Google takes other measures to prevent scraping I will remove the scraper because I'm not going to play chicken with Google.
3bae1cf1