Maximum size of input

fervlrmfervlrm Member Posts: 2 Contributor I
edited November 2018 in Help
Hi all,

I am learning on rapid miner but I would like to know if it will be able to handle a source CSV file with 30 million entries, containing each 26 attributes.... Can rapidminer handle it?



  • Options
    fervlrmfervlrm Member Posts: 2 Contributor I
    In fact,

    I have tried to use the ExampleSetGenerator to generate 27.000.000 samples with 26 attributes and it says JavaHeap Memory error.....
    any solution?
  • Options
    vijaypshahvijaypshah Member Posts: 30 Maven
    Simple Solution: Use 64 bit machine and increase the RAM memory..

    I know matlab and IDL have file association with variable that allows to read only the required part of the file, I am not sure if Java supports it. May be you might want to research on that.

  • Options
    IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder

    yes, increasing the available memory is certainly an option. Another option is to store the data in a database and directly work on it with the appropriate settings.

Sign In or Register to comment.