"Web mining a list of URLs"

adommnichadommnich Member Posts: 2 Contributor I
edited May 2019 in Help
Hello,

how can I loop over a list of different URLs and extract the content? The operator Crawl Web has no input but the parameter 'URL'. Is there a way to fill it dynamically using a Loop operator?

Thanks for your attention.

Regards
adommnich
Tagged:

Answers

  • colocolo Member Posts: 236 Maven
    Hi adommnich,

    I would use a "Loop Examples" operator for the URL list (if you read it into an example set before). Inside the loop you can easily extract the current URL with "Extract Macro" and use the value for the "Get Page" operator. If you want to crawl and not just retrieve one website you can replace "Get Page" by "Crawl Web".

    Regards,
    Matthias
  • adommnichadommnich Member Posts: 2 Contributor I
    Hi Matthias,

    thanks a lot for your quick reply. I am afraid I wasn' t aware of the Macro concept.  :-[

    It works fine now.

    Regards
    Alex
Sign In or Register to comment.