Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

Maximum size of input

fervlrmfervlrm Member Posts: 2 Contributor I
edited November 2018 in Help
Hi all,

I am learning on rapid miner but I would like to know if it will be able to handle a source CSV file with 30 million entries, containing each 26 attributes.... Can rapidminer handle it?

Thanks

Answers

  • fervlrmfervlrm Member Posts: 2 Contributor I
    In fact,

    I have tried to use the ExampleSetGenerator to generate 27.000.000 samples with 26 attributes and it says JavaHeap Memory error.....
    any solution?
  • vijaypshahvijaypshah Member Posts: 30 Maven
    Hi,
    Simple Solution: Use 64 bit machine and increase the RAM memory..

    I know matlab and IDL have file association with variable that allows to read only the required part of the file, I am not sure if Java supports it. May be you might want to research on that.

    Regards,
    Vijay
  • IngoRMIngoRM Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder
    Hi,

    yes, increasing the available memory is certainly an option. Another option is to store the data in a database and directly work on it with the appropriate settings.

    Cheers,
    Ingo
Sign In or Register to comment.