Maximum size of input

fervlrmfervlrm Member Posts: 2 Contributor I
edited November 2018 in Help
Hi all,

I am learning on rapid miner but I would like to know if it will be able to handle a source CSV file with 30 million entries, containing each 26 attributes.... Can rapidminer handle it?

Thanks

Answers

  • fervlrmfervlrm Member Posts: 2 Contributor I
    In fact,

    I have tried to use the ExampleSetGenerator to generate 27.000.000 samples with 26 attributes and it says JavaHeap Memory error.....
    any solution?
  • vijaypshahvijaypshah Member Posts: 30  Guru
    Hi,
    Simple Solution: Use 64 bit machine and increase the RAM memory..

    I know matlab and IDL have file association with variable that allows to read only the required part of the file, I am not sure if Java supports it. May be you might want to research on that.

    Regards,
    Vijay
  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,642  RM Founder
    Hi,

    yes, increasing the available memory is certainly an option. Another option is to store the data in a database and directly work on it with the appropriate settings.

    Cheers,
    Ingo
Sign In or Register to comment.