RapidMiner

RapidMiner

Maximum size of input

Contributor

Maximum size of input

Hi all,

I am learning on rapid miner but I would like to know if it will be able to handle a source CSV file with 30 million entries, containing each 26 attributes.... Can rapidminer handle it?

Thanks
3 REPLIES
Contributor

Re: Maximum size of input

In fact,

I have tried to use the ExampleSetGenerator to generate 27.000.000 samples with 26 attributes and it says JavaHeap Memory error.....
any solution?
Regular Contributor

Re: Maximum size of input

Hi,
Simple Solution: Use 64 bit machine and increase the RAM memory..

I know matlab and IDL have file association with variable that allows to read only the required part of the file, I am not sure if Java supports it. May be you might want to research on that.

Regards,
Vijay
RMStaff

Re: Maximum size of input

Hi,

yes, increasing the available memory is certainly an option. Another option is to store the data in a database and directly work on it with the appropriate settings.

Cheers,
Ingo