Memory problems

vgpowered

Why does Rapidminer stop and show the message "This process would need more than the maximun amount of available memory..."
I know my process needs a lot of memory because it uses a repository data of 5GB but my computer has 32GB and at the time of the fail java.exe uses 9GB an the system has aditional 13.5GB available.
Whats is happening?

rjones13

Hi @vgpowered,

There's a few things to consider here. The first would be the resources you've set in the preferences. If you go to Settings > Preferences > System, you can increase the maximum amount of memory allowed to the system. If you increase this, you should be able to utilise more of your system. 

You could also consider, if appropriate, performing your operations on subsets of the data and looping to keep the process automated.

Let me know if this helps.

Best,
Roland

vgpowered

Thanks in advance. We are still testing your proposal, but it seems to be a memory leak on the Rapidminer platform because the memory in use keeps increasing continuously.

MartinLiebig

Hi,

why do you think so? It might just be that your algorithm needs more and more memory.

BR,

Martin

vgpowered

Thanks Martin. May be. We are still testing, but there is a clear thing if you run a process, changes some of their arguments and test again and test again the process keeps increasing the memory used continuously. It seems RM does not free memory (unless it is results history, in this case RM should think about transferring these results to a file)

MartinLiebig

Hey,

first: If this is a commercial opportunity please check in with your sales rep, so we can investigate this on a more detailed level.

Then: Keep in mind how the JVM works. The JVM is configured to take more an more memory from your OS. It only runs a garbage collector run, if there is a need for it. This means it is increasing and increasing, until it hits the configured ceiling. Only then not needed objects are deleted.

BR,

Martin