"Web crawl - lightbox content / overlay"
If I use the crawl operator I can get the urls that trigger the overlays in a web browser session but saving the html source to a directory doesn't include the overlay and if you crawl the urls directly they only show the base webpage even though each url is different (no overlaid content shown if you visit the urls directly which seems like a bad user experience to me )
If I use selenium running in firefox and Click and wait the overlay is displayed and the source code can be manually saved.
Is it possible to get rapidminer to handle this use case or is it due to some clever coding by the web designers a step too far for rapidminer?
If its not possible to use the native operators whats the best way to integrate selenium code into a rapidminer process to pass back the html code that includes the overlay?