Options
Web Mining, Crawl Web crawling rules...please explain?
Contributor II
I used RapidMiner in my MBA program and it's been almost three years since I last touched it. I just started a position where I'll be using it again and I'm a bit rusty. I'm trying to scrape a site for some data (names, phone numbers, addresses, etc.) and put them into an excel file, however I'm not able to figure out the parameters. I think my main issue is understanding what the crawling rules are. What do they mean? Which should I be applying? I've Googled this and searched here, but I only get instructions specific to other users' questions. Can anyone provide a definition of what these are and what they mean/do?
Tagged:
2
Best Answer
-
Options
kayman
Member Posts: 662 
Unicorn
@Cash , The traditional components won't work here, as this is a dynamic page loading a JSON file with all the locations separately.
So what you will crawl and store is the skeleton only, containing the placeholders where the data will be injected during rendering.
So this requires a bit of reverse engineering, I'll give you some tips but have to state that this might be on the borderline of what is ethical crawling.
If you load the page in for instance firefox with the inspect element window open (shortcut Q on windows) and select the network tab you can see where this page get's all its content from. This goes from images over scripts etc, and one of the sources is a rather large json file called from an API, that seems to have all the locations.
So purely in theory you can download this json file directly if the site owner has no problems with this, and use JSON to Data to deal with it from there.6
Answers
Take a look at this link please
https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_web
I hope this helps
mbs
I'm just trying to capture the names, locations, and phone numbers. I used Selector Gadget to help me figure out the CSS tags I need and this is what it has given me: .places-app-location-citystatezip , a , .places-app-location-street , .places-app-location-name