Answers

• Options

  varunm1 Moderator, Member Posts: 1,207 Unicorn

  December 2018

  Hi Alex, I set it manually to 20 GB but the task manager shows memory greater than 20 GB for the RM process. I understood the issue based on your explanation as this cannot be avoided with GUI usage.

  Regards,
  Varun
  https://www.varunmandalapu.com/
  
  

  Be Safe. Follow precautions and Maintain Social Distancing

  0
• Options

  SGolbert RapidMiner Certified Analyst, Member Posts: 344 Unicorn

  January 2019

  Hi Varun,

  
  

  can you load the dataset into RapidMiner (Retrieve without further operators, run and see results) and see how is the memory consumption? I could be that the RM table is using memory for all the missing values, in which case the dataset is very large. In that case you have to use the Declare Missing Values operator and make sure that RM sees them as missing and not as an empty string for example.

  
  

  Regards,

  Sebastian
  

  
  

  
  

  0
• Options

  varunm1 Moderator, Member Posts: 1,207 Unicorn

  January 2019

  Hi Sebastian,
  
  Thanks for your response. The RM GUI is taking 3.5 GB approx with a blank process window. I ran the dataset as mentioned by you and its closer to 5 GB. The dataset doesn't have missing values as it is generated by another algorithm in Matlab. Please see screenshots below for your reference.
  
  Initial GUI:
  
  
  
  While data is getting processed:
  
  
  Thanks,
  Varun

  Regards,
  Varun
  https://www.varunmandalapu.com/
  
  

  Be Safe. Follow precautions and Maintain Social Distancing

  0
• Options

  David_A Administrator, Moderator, Employee, RMResearcher, Member Posts: 297 RM Research

  January 2019

  Hi,
  
  don't be confused about the RAM usage for an empty process. Java tends to allocate a lot of memory in advance, that's what you see there.
  
  Are you using the Deep Learning operator from core RapidMiner (with the H2O library) or the Deep Learning extension?

  2
• Options

  varunm1 Moderator, Member Posts: 1,207 Unicorn

  January 2019

  Hi @David_A
  
  Deep Learning Extension. I also got to know from @hughesfleming68 there are some memory leaks which he got to your notice.
  
  Thanks
  Varun

  Regards,
  Varun
  https://www.varunmandalapu.com/
  
  

  Be Safe. Follow precautions and Maintain Social Distancing

  0
• Options

  hughesfleming68 Member Posts: 323 Unicorn

  January 2019 edited January 2019

  Hi Varun, those might be fixed as they were for an earlier version of RM9. It was one process that was using 40G of memory. There were actually over 500 operators in that process. It worked correctly on 8.2 but I have not tested it again. I will check over the weekend. In the end, I duplicated the whole process in python and I have been using it that way ever since.

  
  

  Edit: I just tested it again and it worked fine in 9.1. Still used 37G but didn't crash. It was fast as well, faster than I remember.
  

  1
• Options

  SGolbert RapidMiner Certified Analyst, Member Posts: 344 Unicorn

  January 2019

  Hi @varunm1,

  
  

  I mentioned the missing values, because RapidMiner has different internal structures to represent example sets. One of them is optimized for sparse matrices, but if the values doesn't appear as missing to RapidMiner (even though they were missing in some other program or database), this data structure won't be loaded. It's just a hunch, as I don't know the internal functions in detail.
  

  
  

  Another question is whether the extensions can make use of these optimized data structures.

  
  

  Regards,

  Sebastian
  

  1