We have TONS of videos to help you learn RapidMiner - from beginner to expert. Click to watch!
We're competing as Team "RapidMiners" in DrivenData's latest competition: "Pover-T". Join now!
Read about how our community works. Meet other newbies. Get your questions answered fast!
There are many pages on the web that contain useful data in the form of simple html tables. Here's an example:
I know how to use RapidMiner to retrieve this data automatically in html form using "get page" and store it as a document, and I even know how to do this iteratively if a set of related pages are required. I also have some familiarity with how to manipulate documents, but what I really want is to extract the information in the html table into a usable example set in RapidMiner. Is there any relatively simple way of doing the following:
It seems like it would be an incredibly useful operator that did all this automatically - "HTML table to data" or something similar. I'm fairly certain that such an operator doesn't exist (yet), but I'm not even sure of the collection of existing operators that would be required to do all of the above. Any ideas @mschmitz or @Thomas_Ott ?
Unfortunately there is no HTML table to Example set operator. I wish there was because there is much good info trapped in tables around the Interwebz. The only work around I can see is using Xpath to extract the tables from their respective <div> and <td>, <tr> tags.
@mschmitz any other ideas?
I believe that is basically the xpath route that @Thomas_Ott is suggesting--which unfortunately involves a lot of manual fiddling with xpath expressions. Plus I think you would actually need two separate streams, one to extract the column titles and apply those as attribute names, and another to extract the actual data for the examples. So it's all possible, just not easy (and probably not for the faint of heart either). Well, I know what I will be suggesting next in the product ideas forum :-)
@Thomas_Ott If anyone comes up with an elegant solution to this, I am still interested. My experience is that the HTML to XML is still quite messy. In the meantime, my preferred workaround is using Google Sheets to read the html table from the URL (they have a very handy function for exactly that, much like what I was suggesting for RapidMiner). Then I can export that sheet as a csv and read it into RapidMiner. It would be even nicer if RapidMiner had the ability to dynamically read Google Sheets, but I haven't figured that out yet either...
Hmm, I recently did a proof of concept where I imported a RM webservice into Google Sheets and managed to score data in the Google Sheet by passing to to the Server.
I wonder if there's a way to cut out that export step you're using.