The maximum amount of memory RapidMiner Server uses is specified in a settings file and has to be changed by hand.
To manually alter the maximum amount of memory an already installed RapidMiner Server uses, follow these steps:
Open the bin/standalone.conf.bat file in your RapidMiner Server installation directory.
Search for the line which contains the following fragment (number might differ) and replace the number with the amount of memory you want the server to use (in megabytes):
Restart RapidMiner Server.
On Linux / Mac:
Open the bin/standalone.conf file in your RapidMiner Server installation directory.
I believe this is a bit outdated, I also had to modify the max-memory.conf file in ./standalone/configuration before the actual values were changed.