I have tried almost everything but I seem to be stuck with a Java maximum memory of 1.5 Gig RAM though I tried on different computers (both windows 2000 and windows xp) with 3 Gig Ram. It doesn't help to increase the variable in the .bat - file nor in the java console - it always tells me cannot create Virtual Memory... if I enter anything above 1.5 Gig.
so the maximum I always end up with is 1.4 Gig (as it says in the GUI) The process I want to run is a Weka-Adaboost, admittedly the training-data size is quite large (more than 2000000 rows and 25 attributes) but I tried the same process with a smaller training size (500000 and 25 attributes) and it worked ok.
So is there ANY way to convince this stupid JVM to use all my RAM??????
below is the log-file of my process:
G Mar 3, 2009 2:19:12 PM: Checking properties...
G Mar 3, 2009 2:19:12 PM: Properties are ok.
G Mar 3, 2009 2:19:12 PM: Checking process setup...
G Mar 3, 2009 2:19:12 PM: Inner operators are ok.
G Mar 3, 2009 2:19:12 PM: Checking i/o classes...
G Mar 3, 2009 2:19:12 PM: i/o classes are ok. Process output: Model.
G Mar 3, 2009 2:19:12 PM: Process ok.
P Mar 3, 2009 2:19:12 PM: Logging: log file is 'ada_log.log'...
P Mar 3, 2009 2:19:12 PM: Initialising process setup
P Mar 3, 2009 2:19:12 PM: Checking properties...
P Mar 3, 2009 2:19:12 PM: Properties are ok.
P Mar 3, 2009 2:19:12 PM: Checking process setup...
P Mar 3, 2009 2:19:12 PM: Inner operators are ok.
P Mar 3, 2009 2:19:12 PM: Checking i/o classes...
P Mar 3, 2009 2:19:12 PM: i/o classes are ok. Process output: Model.
P Mar 3, 2009 2:19:12 PM: Process ok.
P Mar 3, 2009 2:19:13 PM: Process initialised
P Mar 3, 2009 2:19:13 PM: [NOTE] Process starts
P Mar 3, 2009 2:19:13 PM: Process:
+- ExampleSource (ExampleSource)
+- W-AdaBoostM1 (W-AdaBoostM1)
| +- W-REPTree (W-REPTree)
+- ModelWriter (ModelWriter)
G Mar 3, 2009 2:27:48 PM: [Fatal] OutOfMemoryError occured in 1st application of W-AdaBoostM1 (W-AdaBoostM1)
G Mar 3, 2009 2:27:48 PM: [Fatal] Process failed: Java heap space
In short, it's a WIndows problem, not a Java or RapidMiner problem.
The link I posted explains it in detail, but it summarized by: "On Windows, by default, the lower 2 GB are reserved for user-mode programs and the upper 2 GB are reserved for kernel-mode programs."
Basically, when you run a 32-bit Windows operating system (as I assume you are), regardless of the amount of RAM you have installed, by default, it will only allocate a maximum of 2GB to an application (reserving the other 2GB for the OS). With Java running, it basically means only 1.5GB is actually available to RapidMiner.
The switch changes the operating system allocation to allot 3 GB to applications (and 1 GB for the OS), so you may be able to run RM with more memory.
OK, I just checked on my own system, and the /3GB switch allowed me to get a little more memory allocated to RM (about 1.4 GB), but not as much as I had thought. A bit of Googling explained why.
Basically, Java requires heap space to be allocated in a single contiguous block of memory, but Windows 32-bit address space is fragmented, and can't allocate a single block of memory larger than about 1.5GB, even with the /3GB switch. In other words, Windows 32-bit can't give the memory to Java the way Java needs it.
There are other applications (e.g. I've done it with "R" statistical software), which can deal with multiple chunks of memory, and can take advantage of the /3GB to allocate a total of about 2.8GB of RAM. But anything Java-based such as RapidMiner is going to have the same problem you've encountered here. The best solution is to upgrade to a 64-bit version of Windows or Linux, which doesn't have the memory limitations of 32-bit to contend with.
Hi Marcello, this seems to be very interesting. Have you already performed some reliable tests? I would be very interested in the influence of the different systems and JVM settings. Do you have a nice setup there?