Options

Crawl web operator does not return any results

AmiroAmiro Member Posts: 3 Newbie
edited January 2019 in Help
Hi I am having issues with the simplest web crawler.  It does not return any results at all. I have no rules, just a simple web crawl out of the box. What am I doing wrong here?



Tagged:

Answers

  • Options
    AmiroAmiro Member Posts: 3 Newbie



  • Options
    AmiroAmiro Member Posts: 3 Newbie
    I have attached my rmp file, any help would be appreciated? Many thanks
  • Options
    sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    hi @Amiro - so that website does not have much content and it is a very heavily-stylized site. My guess is that the site is blocking very basic crawlers like "Crawl Web". You're better off using Get Page, grabbing URLs, and so on.

    Scott

  • Options
    Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    Or it could be related to the https problem that came up in the other thread as well.  At this point I think Crawl Web operator is not a totally reliable option. Using Get Page within some kind of loop is probably a better alternative.
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
Sign In or Register to comment.