How are you starting RapidMiner? Using the .exe, this should not happen. If you run it by invoking java directly, you will have to set the heap size using the java -Xm switch. You can see how this is done if you look at the start scripts in the scripts folder. You can edit the MAX_JAVA_MEMORY variable there.