Crawl Google Search

AtiahKhoirunnisa

Hello everyone,

Can Rapid Miner crawl documents from Google Search, example i want to retrieve all the document that related to what i search in Google search, example i wanna to find docs or headlines that contain  'Obama', how do  i do that? 

SGolbert

There are ways to do it, but Google Searches are not legally crawlbable:

Check out:

https://www.google.com/robots.txt

If you still want to do it with smaller chances of getting banned or having problems, you need to use Selenium, a web crawler that uses your web browser (and is therefore quite slow).

Kind regards,

Sebastian

MarcoBarradas

@AtiahKhoirunnisa
You could use the Custom Search Engine and retrieve the information from the JSON
https://cse.google.com/cse/
Keep in mind that their is a limit of places you can search information from and if you need to increase the amount of information you may need you'll need to pay for the usage of the tool.
But I guess this could help you.
Best Regards

AtiahKhoirunnisa

Thank you Golbert and Marco, I will try first