How to download dynamic information

gortipavangortipavan Member Posts: 1 Contributor I
edited November 2018 in Help
Hi

I have a problem at hand and am wondering if RapidMiner can be used to solve it.

I need to get some information from a website. It is a website that reports scientific information for different lakes. I would like to fetch all the data corresponding to a lake attribute (example, Lake depth). This data is loaded on the website dynamically and is not part of the initial markup (The database is read to dynamically populate certain fields). Is tehre a way I can mine this information using Rapid miner? If so, can someone point me in the right direction?

Thanks

Answers

  • el_chiefel_chief Member Posts: 63 Contributor II
    scraping websites that use javascript can be difficult.

    i would recommend looking at Selenium, which lets you use say Chrome browser to fetch a web page and create the markup for you, and return that as text.

    http://seleniumhq.org/

    you could also try the watij library which is a java library for scraping sites with dynamic content

    http://watij.com/

    regards,

    neil
Sign In or Register to comment.