RapidMiner

RapidMiner

Web mining a list of URLs

Contributor

Web mining a list of URLs

Hello,

how can I loop over a list of different URLs and extract the content? The operator Crawl Web has no input but the parameter 'URL'. Is there a way to fill it dynamically using a Loop operator?

Thanks for your attention.

Regards
adommnich
2 REPLIES
Regular Contributor

Re: Web mining a list of URLs

Hi adommnich,

I would use a "Loop Examples" operator for the URL list (if you read it into an example set before). Inside the loop you can easily extract the current URL with "Extract Macro" and use the value for the "Get Page" operator. If you want to crawl and not just retrieve one website you can replace "Get Page" by "Crawl Web".

Regards,
Matthias
Contributor

Re: Web mining a list of URLs

Hi Matthias,

thanks a lot for your quick reply. I am afraid I wasn' t aware of the Macro concept.  :-[

It works fine now.

Regards
Alex