The RapidMiner community is on read-only mode until further notice. Technical support via cases will continue to work as is. For any urgent licensing related requests from Students/Faculty members, please use the Altair academic forum here.
Need to crawl webpages requiring login details
Hello,
i need to crawl certain websites but they require login details to be entered.
I am not able to figure out how to provide my login details in order to get access to my homepage.Is there any operator or any other way to do that??
Please help.!!
Thanks and Regards,
Vineet
i need to crawl certain websites but they require login details to be entered.
I am not able to figure out how to provide my login details in order to get access to my homepage.Is there any operator or any other way to do that??
Please help.!!
Thanks and Regards,
Vineet
0
Answers
In the Get-Page operator is an option to activate cookies. Activate it and send your credentials (username and password) to the login page as POST parameter. Usually a web page will store your session in a cookie. Further requests of any Get-Page(s) operator will be handled by the website in the same session (using the stored cookies), thus you are logged in (if your credentials are correct) and can fetch the login-secured websites.
Happy crawling!
Marcin
Appreciate your quick reply. It was the same i was trying to do. But somehow i am not able to do it.
Here is my code for your reference.
Please tell me where am i going wrong.
Thanks and Regards,
Vineet