"Can I use a macro value to crawl sites ?"

kayman

Hi there,

Can I use macro values one way or another with the Crawl web operator to set the url value ? I've tried the obvious %{my_macro_value} but the system didn't like that.

I would like to use this option (or any alternative as my process is as follows :

I have an example set containing urls I want to crawl, for example

www.somedomain.com/somepageformat?id=12345
www.somedomain.com/somepageformat?id=54321
www.somedomain.com/somepageformat?id=...
...

However, every url above can have several pages also, like

www.somedomain.com/somepageformat?id=12345&;page=1
www.somedomain.com/somepageformat?id=12345&;page=2

which I all need to crawl strting from the 'master' page. Works perfectly if I use the crawl web parameter with static data, but since the list is variable I need to find a way to provide the id, or the full url, as a variable to the operator. Using the get page operator is not a real option since I do not know upfront how many pages are available as query string.

Any suggestion ?