Rapidminer to HPC

icsdm16069

Hello,

i would like to ask if i could run the rapidminer (using linux distribution) in an HPC with slurm. I am really confused because i am trying to run it and have memory problems. My rapidminer process is working well and is inside a loop. Is anyone that have the same problem?

MartinLiebig

Hi @icsdm16069 ,

how much ram did you give it?

BR,

Martin

icsdm16069

my slurm program is the following:

#!/bin/bash -l

####################################
#      slurm script template   #
#                                  #
# Submit script: sbatch filename   #
#                                  #
####################################

#SBATCH --job-name=rapidminer    # Job name
#SBATCH --output=rapidminer.%j.out # Stdout (%j expands to jobId)
#SBATCH --error=rapidminer.%j.err # Stderr (%j expands to jobId)
#SBATCH --ntasks=3     # Number of tasks(processes)
#SBATCH --nodes=1     # Number of nodes requested
#SBATCH --ntasks-per-node=3     # Tasks per node
#SBATCH --cpus-per-task=1     # Threads per task
#SBATCH --time=10:10:30   # walltime
#SBATCH --mem=64G   # memory per NODE
#SBATCH --partition=task    # Partition
#SBATCH --account=pa    # Replace with your system project

if [ x$SLURM_CPUS_PER_TASK == x ]; then
  export OMP_NUM_THREADS=1
else
  export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
fi


## LOAD MODULES ##
module purge		# clean up loaded modules 

# load necessary modules
module load gnu/4.9.2
module load intel/15.0.3
module load intelmpi/5.0.3
module load cuda/8.0.61
module load java/1.8.0
module load python/3.5.0
## RUN YOUR PROGRAM ##
srun ./rapidminer-studio/scripts/rapidminer-batch.sh -f /rapidminercodes/xml.rmp

icsdm16069

ok i fixed it. The first thing that you need to modify is the max memory that can be used from rapidminer and the second is the making an mpi program in order to split your input data and run the rapidminer in parallel.