Web mining, get numerical data through going through provided calendar

felix_wfelix_w Member Posts: 61 Contributor II
edited September 2019 in Help

Dear Rapidminer Community, 


only recently I discovered that RM also offers webmining operators which is of great help for me! 

But unfortunately I encountered a problem when I tried to download data from this webpage 


I would like to download all hourly data 00-01 etc for a specific time (e.g. 01.01.2015 - 30.09.2017) but I can only do this through manually changing the dates day per day in the provided calendar on the website, which is not very handy. 


My question now, is there any way in RM to configure the Webcrawler to go through the calender automatically and draw the hourly data?


I hope you can help me with my question. 


Best regards




  • Options
    Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,761 Unicorn

    This might be tricky indeed. I don't think the webcrawler from RapidMiner can do this and you might have to resort to some outside script to do it (like node.js).  You can use the Execute Program operator to call the script, change the date, load the new page and then run the RM crawler. 

  • Options
    felix_wfelix_w Member Posts: 61 Contributor II

    Thank you for your reply! I thought already that this could be slightly tricky ;)

Sign In or Register to comment.