Presence of large stores makes RM not responsive
jacobcybulski
Member, University Professor Posts: 391 
Unicorn
I have a project with some very large data stores (converted from Kaggle CSVs). I find that on Ubuntu 16.04 the presence of these stores does not cause any issues. However on Win10 PCs RM cannot even open the folder with these stores without a few minutes delay. On a small laptop, as a result Java takes 10Gb of RAM and never completes the task - clearly the stores are not being loaded into RAM just to open their folder? On occasions, it seems RM is busy "Validating Process" (the green indicator bottom-right of the window). I found a work-around to use these large data stores by placing them in a separate folder and never touch the folder itself from within RM. It is quite OK to use those stores by referring to them in a separate folder. There are no errors in a log and no Window errors. Any ideas?
Answers
my gut instinct question is how much RAM is on your laptop?
Also cc'ing @jczogalla @Marco_Boeck in case they have further thoughts...
Scott
Was there any additional information about this issue after the original post? My own suspicion is that when opening the folder, RM is trying to access all the contents' metadata in the background, which might actually be quite large if the underlying repositories are large. Whereas if you don't directly open the folder, RM just leaves it alone and thus doesn't slow anything down. But I am not a RapidMiner developer so I don't really know what is going on in the background, this is just my hypothesis after observations.
Lindon Ventures
Data Science Consulting from Certified RapidMiner Experts