Options

Memory problems

vgpoweredvgpowered Member Posts: 30 Contributor II
Why does Rapidminer stop and show the message "This process would need more than the maximun amount of available memory..."
I know my process needs a lot of memory because it uses a repository data of 5GB but my computer has 32GB and at the time of the fail java.exe uses 9GB an the system has aditional 13.5GB available.
Whats is happening?

Answers

  • Options
    rjones13rjones13 Member Posts: 180 Unicorn
    Hi @vgpowered,

    There's a few things to consider here. The first would be the resources you've set in the preferences. If you go to Settings > Preferences > System, you can increase the maximum amount of memory allowed to the system. If you increase this, you should be able to utilise more of your system. 

    You could also consider, if appropriate, performing your operations on subsets of the data and looping to keep the process automated.

    Let me know if this helps.

    Best,
    Roland
  • Options
    vgpoweredvgpowered Member Posts: 30 Contributor II
    Thanks in advance. We are still testing your proposal, but it seems to be a memory leak on the Rapidminer platform because the memory in use keeps increasing continuously.
  • Options
    MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,520 RM Data Scientist
    Hi,
    why do you think so? It might just be that your algorithm needs more and more memory.

    BR,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Options
    vgpoweredvgpowered Member Posts: 30 Contributor II
    Thanks Martin. May be. We are still testing, but there is a clear thing if you run a process, changes some of their arguments and test again and test again the process keeps increasing the memory used continuously. It seems RM does not free memory (unless it is results history, in this case RM should think about transferring these results to a file)
  • Options
    MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,520 RM Data Scientist
    Hey,
    first: If this is a commercial opportunity please check in with your sales rep, so we can investigate this on a more detailed level.

    Then: Keep in mind how the JVM works. The JVM is configured to take more an more memory from your OS. It only runs a garbage collector run, if there is a need for it. This means it is increasing and increasing, until it hits the configured ceiling. Only then not needed objects are deleted.

    BR,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
Sign In or Register to comment.