Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

Webcrawler download images?

mrbigglesworthmrbigglesworth Member Posts: 1 Learner III
edited November 2018 in Help
Hi,

I haven't posted on this forum before but I've been using Rapidminer recently to do webcrawling and run some analysis on the websites. In certain cases, I would like to use rapidminer's webcrawler save the full page rather than just the html. Specifically, the page may have some jpg files, and I'd like to archive those in certain cases. Is there an easy way to do this (other than writing a custom groovy script)?

Thanks!

PS - Thank you very much for the great support and great program. Rapidminer keeps getting better.

Answers

  • colocolo Member Posts: 236 Maven
    Hi,

    as far as I know the embedded WebSphinx crawler doesn't support storing additional media. But better check the documentation to be sure: http://www.cs.cmu.edu/~rcm/websphinx/

    There was a discussion about replacing this ancient crawler some time ago. I would also be glad if a better crawler was available in RapidMiner, but since there is one that is working certainly other things have priority. Maybe I will integrate another crawler after hopefully finishing my thesis this month, if I start missing the development of RapidMiner add-ons ;)

    Regards
    Matthias
Sign In or Register to comment.