Options

Rapidminer to HPC

icsdm16069icsdm16069 Member Posts: 3 Contributor I
Hello,

i would like to ask if i could run the rapidminer (using linux distribution) in an HPC with slurm. I am really confused because i am trying to run it and have memory problems. My rapidminer process is working well and is inside a loop. Is anyone that have the same problem?

Answers

  • Options
    MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,517 RM Data Scientist
    how much ram did you give it?
    BR,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Options
    icsdm16069icsdm16069 Member Posts: 3 Contributor I
    my slurm program is the following:

    #!/bin/bash -l
    
    ####################################
    #      slurm script template   #
    #                                  #
    # Submit script: sbatch filename   #
    #                                  #
    ####################################
    
    #SBATCH --job-name=rapidminer    # Job name
    #SBATCH --output=rapidminer.%j.out # Stdout (%j expands to jobId)
    #SBATCH --error=rapidminer.%j.err # Stderr (%j expands to jobId)
    #SBATCH --ntasks=3     # Number of tasks(processes)
    #SBATCH --nodes=1     # Number of nodes requested
    #SBATCH --ntasks-per-node=3     # Tasks per node
    #SBATCH --cpus-per-task=1     # Threads per task
    #SBATCH --time=10:10:30   # walltime
    #SBATCH --mem=64G   # memory per NODE
    #SBATCH --partition=task    # Partition
    #SBATCH --account=pa    # Replace with your system project
    
    if [ x$SLURM_CPUS_PER_TASK == x ]; then
      export OMP_NUM_THREADS=1
    else
      export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
    fi
    
    
    ## LOAD MODULES ##
    module purge		# clean up loaded modules 
    
    # load necessary modules
    module load gnu/4.9.2
    module load intel/15.0.3
    module load intelmpi/5.0.3
    module load cuda/8.0.61
    module load java/1.8.0
    module load python/3.5.0
    ## RUN YOUR PROGRAM ##
    srun ./rapidminer-studio/scripts/rapidminer-batch.sh -f /rapidminercodes/xml.rmp

  • Options
    icsdm16069icsdm16069 Member Posts: 3 Contributor I
    ok i fixed it. The first thing that you need to modify is the max memory that can be used from rapidminer and the second is the making an mpi program in order to split your input data and run the rapidminer in parallel.
Sign In or Register to comment.