ANNOUNCEMENT: RAPIDMINER 9.1 HAS JUST BEEN RELEASED!   PLEASE DOWNLOAD AND GIVE FEEDBACK. ENJOY AND HAPPY RAPIDMINING!   -- @sgenzer – Community Manager

Crawl web operator does not return any results

AmiroAmiro Member Posts: 3 Newbie
Hi I am having issues with the simplest web crawler.  It does not return any results at all. I have no rules, just a simple web crawl out of the box. What am I doing wrong here?



Answers

  • AmiroAmiro Member Posts: 3 Newbie
  • AmiroAmiro Member Posts: 3 Newbie
    I have attached my rmp file, any help would be appreciated? Many thanks
  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager Posts: 1,888  Community Manager
    hi @Amiro - so that website does not have much content and it is a very heavily-stylized site. My guess is that the site is blocking very basic crawlers like "Crawl Web". You're better off using Get Page, grabbing URLs, and so on.

    Scott

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 882   Unicorn
    Or it could be related to the https problem that came up in the other thread as well.  At this point I think Crawl Web operator is not a totally reliable option. Using Get Page within some kind of loop is probably a better alternative.
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
    sgenzer
Sign In or Register to comment.