RapidMiner

RapidMiner

Java Heap Space

ada
Contributor II

Java Heap Space

Dear specialists,

I have tried almost everything but I seem to be stuck with a Java maximum memory of 1.5 Gig RAM though I tried on different computers (both windows 2000 and windows xp) with 3 Gig Ram.
It doesn't help to increase the variable in the .bat - file nor in the java console - it always tells me cannot create Virtual Memory... if I enter anything above 1.5 Gig.

so the maximum I always end up with is 1.4 Gig (as it says in the GUI)
The process I want to run is a Weka-Adaboost, admittedly the training-data size is quite large (more than 2000000 rows and 25 attributes) but I tried the same process with a smaller training size (500000 and 25 attributes) and it worked ok.

So is there ANY way to convince this stupid JVM to use all my RAM??????

below is the log-file of my process:


G Mar 3, 2009 2:19:12 PM: Checking properties...

G Mar 3, 2009 2:19:12 PM: Properties are ok.

G Mar 3, 2009 2:19:12 PM: Checking process setup...

G Mar 3, 2009 2:19:12 PM: Inner operators are ok.

G Mar 3, 2009 2:19:12 PM: Checking i/o classes...

G Mar 3, 2009 2:19:12 PM: i/o classes are ok. Process output: Model.

G Mar 3, 2009 2:19:12 PM: Process ok.

P Mar 3, 2009 2:19:12 PM: Logging: log file is 'ada_log.log'...

P Mar 3, 2009 2:19:12 PM: Initialising process setup

P Mar 3, 2009 2:19:12 PM: Checking properties...

P Mar 3, 2009 2:19:12 PM: Properties are ok.

P Mar 3, 2009 2:19:12 PM: Checking process setup...

P Mar 3, 2009 2:19:12 PM: Inner operators are ok.

P Mar 3, 2009 2:19:12 PM: Checking i/o classes...

P Mar 3, 2009 2:19:12 PM: i/o classes are ok. Process output: Model.

P Mar 3, 2009 2:19:12 PM: Process ok.

P Mar 3, 2009 2:19:13 PM: Process initialised

P Mar 3, 2009 2:19:13 PM: [NOTE] Process starts

P Mar 3, 2009 2:19:13 PM: Process:

  Root[0] (Process)

  +- ExampleSource[0] (ExampleSource)

  +- W-AdaBoostM1[0] (W-AdaBoostM1)

  |  +- W-REPTree[0] (W-REPTree)

  +- ModelWriter[0] (ModelWriter)

G Mar 3, 2009 2:27:48 PM: [Fatal] OutOfMemoryError occured in 1st application of W-AdaBoostM1 (W-AdaBoostM1)

G Mar 3, 2009 2:27:48 PM: [Fatal] Process failed: Java heap space

          Root[1] (Process)

          +- ExampleSource[1] (ExampleSource)

here ==> +- W-AdaBoostM1[1] (W-AdaBoostM1)

          |  +- W-REPTree[0] (W-REPTree)

          +- ModelWriter[0] (ModelWriter)

10 REPLIES
Regular Contributor

Re: Java Heap Space

Are you booting XP/2000 with the /3GB boot parameter?

http://msdn.microsoft.com/en-us/library/ms791558.aspx
ada
Contributor II

Re: Java Heap Space

thanks for the reply!

but no I don't, why should I?

I have more the 3.4 Gig RAM available in my system and those I want the JVM to use.

would it help if I did this boot thing?
Regular Contributor

Re: Java Heap Space

In short, it's a WIndows problem, not a Java or RapidMiner problem. 

The link I posted explains it in detail, but it summarized by:  "On Windows, by default, the lower 2 GB are reserved for user-mode programs and the upper 2 GB are reserved for kernel-mode programs."

Basically, when you run a 32-bit Windows operating system (as I assume you are), regardless of the amount of RAM you have installed, by default, it will only allocate a maximum of 2GB to an application (reserving the other 2GB for the OS).  With Java running, it basically means only 1.5GB is actually available to RapidMiner.

The switch changes the operating system allocation to allot 3 GB to applications (and 1 GB for the OS), so you may be able to run RM with more memory.

ada
Contributor II

Re: Java Heap Space

I did try it now with the /3GB boot-option but this doesn't change my JVM problem, since it still doesn't allow me to use more than 1.4 GB

is there any way around??? or is there any other java version that allows for a bigger max memory????
Regular Contributor

Re: Java Heap Space

OK, I just checked on my own system, and the /3GB switch allowed me to get a little more memory allocated to RM (about 1.4 GB), but not as much as I had thought.  A bit of Googling explained why.

Basically, Java requires heap space to be allocated in a single contiguous block of memory, but Windows 32-bit address space is fragmented, and can't allocate a single block of memory larger than about 1.5GB, even with the /3GB switch.  In other words, Windows 32-bit can't give the memory to Java the way Java needs it.

A more complete answer is here:  http://blogs.sun.com/moazam/entry/why_can_t_i_allocate

There are other applications (e.g. I've done it with "R" statistical software), which can deal with multiple chunks of memory, and can take advantage of the /3GB to allocate a total of about 2.8GB of RAM.  But anything Java-based such as RapidMiner is going to have the same problem you've encountered here.  The best solution is to upgrade to a 64-bit version of Windows or Linux, which doesn't have the memory limitations of 32-bit to contend with. 
ada
Contributor II

Re: Java Heap Space

thanks a lot for your help!!!

so I'll have to finally upgrade.....grrrr
Regular Contributor

Re: Java Heap Space

Hi Ada,

I think I was having the trouble as you a while ago.  I seem to rember changing to a newer version of java and the problem resolved its self.

Dont know if this will help you but might be worth a try...

Laura
Regular Contributor

Re: Java Heap Space

Hi people,

I was reading this forum and I was interested about this issue.

First of all, windows 32bits has no chance with big dataset and java.

To use all memory with java, as you know, it's necessary use 64bits operational systems and install 64bits jvm (http://java.sun.com/javase/downloads/index.jsp)

Now, windows 64bits don't has the same performance compared to linux 64bits, both with the same hardware.

I my case, I installed Mandriva 64bits, but I recommend Fedora 64bits too over desktops or laptops.

I has a Sony Vaio VGN-FZ470E with with 4GB RAM. I run big dataset to text mining tasks it. Over 2000 attribute and 3000 examples. It's no problem, in 10 to 20 minutes all k-means, EM, SVM, Trees ends.

I learned to optimize my laptop, and as a result, my laptop, literally, fly.  Smiley Surprised ;D

First, if you have 4GB RAM try to put this jvm arguments to starts your RapidMiner:

-Xms3078m -Xmx3078m -Xmn2048m -Xss128k -d64 -server -XXSmiley FrustratedurvivorRatio=8 -XX:+UseParallelGC -XX:+UseParallelOldGC  -XX:+AggressiveOpts -XXSmiley TonguearallelGCThreads=20

THIS PARAMETERS WORK OVER WINDOWS TOO.

References: http://java.sun.com/performance/reference/whitepapers/tuning.html

It's possible running faster, but now you need do the instructions in:
  http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html

After, your jvm arguments should changes to:

-Xms2048m -Xmx2048m -Xmn1024m -Xss128k -d64 -server -XXSmiley FrustratedurvivorRatio=8 -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+UseLargePages -XX:+AggressiveOpts -XXSmiley TonguearallelGCThreads=20

LargePages it's possible only in Linux and Unix based systems (or Windows 2003 Advanced Server, but who has this SO?). It's the difference about windows systems.

With the last changes, my laptop is flying ....  ;D Smiley Surprised

Good luck,
Marcello Sandi
Elite

Re: Java Heap Space

Hi Marcello,
this seems to be very interesting. Have you already performed some reliable tests? I would be very interested in the influence of the different systems and JVM settings. Do you have a nice setup there?

Greetings,
  Sebastian
Old World Computing - Establishing the Future

Check out the Jackhammer Extension for RapidMiner! Crunch more data easier and with up to 700% speed up! Available only here