Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.
Large data set model apply
Hi, I try to apply model on 10mln records database. I use "read database" operator but it copies all data from database to memory in my computer so it coses out of memory exception, moreover there is timeout on database. "Stream database" looks nice but it looks like it works only to make model not to apply (I got an error when applaying with this operator). I think about building a loop to get data with parametrized SQL limit - limiting data f.eg. to 10 000 records is working very well in applying model. Please help - I think there is smarter way than making loops. Most of ETL got streaming DB read.
Thanks
Thanks
0
Answers
using a loop is a perfect work-around if Stream Database does not work for you. As always, posting your process setup and the details of the error message could be useful.
Best, Marius
Apr 11, 2012 1:19:44 PM SEVERE: Process failed: operator cannot be executed. Check the log messages...
Apr 11, 2012 1:19:44 PM SEVERE: Here: Process[1] (Process)
subprocess 'Main Process'
==> +- Stream Database[1] (Stream Database)
+- Write CSV[0] (Write CSV)
Apr 11, 2012 1:19:44 PM SEVERE: java.lang.NullPointerException
with Stream database connected only to CSV output or even to screen: Now I'am testing to export RMiner PMML export and use it in streaming process in Pentaho. I will write how it works. Thanks