Options

Webcrawler download images?

mrbigglesworthmrbigglesworth Member Posts: 1 Contributor I
edited November 2018 in Help
Hi,

I haven't posted on this forum before but I've been using Rapidminer recently to do webcrawling and run some analysis on the websites. In certain cases, I would like to use rapidminer's webcrawler save the full page rather than just the html. Specifically, the page may have some jpg files, and I'd like to archive those in certain cases. Is there an easy way to do this (other than writing a custom groovy script)?

Thanks!

PS - Thank you very much for the great support and great program. Rapidminer keeps getting better.

Answers

  • Options
    colocolo Member Posts: 236 Maven
    Hi,

    as far as I know the embedded WebSphinx crawler doesn't support storing additional media. But better check the documentation to be sure: http://www.cs.cmu.edu/~rcm/websphinx/

    There was a discussion about replacing this ancient crawler some time ago. I would also be glad if a better crawler was available in RapidMiner, but since there is one that is working certainly other things have priority. Maybe I will integrate another crawler after hopefully finishing my thesis this month, if I start missing the development of RapidMiner add-ons ;)

    Regards
    Matthias
Sign In or Register to comment.