Presence of large stores makes RM not responsive

jacobcybulskijacobcybulski Member, University Professor Posts: 391 Unicorn
edited December 2018 in Help

I have a project with some very large data stores (converted from Kaggle CSVs). I find that on Ubuntu 16.04 the presence of these stores does not cause any issues. However on Win10 PCs RM cannot even open the folder with these stores without a few minutes delay. On a small laptop, as a result Java takes 10Gb of RAM and never completes the task - clearly the stores are not being loaded into RAM just to open their folder? On occasions, it seems RM is busy "Validating Process" (the green indicator bottom-right of the window). I found a work-around to use these large data stores by placing them in a separate folder and never touch the folder itself from within RM. It is quite OK to use those stores by referring to them in a separate folder. There are no errors in a log and no Window errors. Any ideas?


  • Options
    sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    my gut instinct question is how much RAM is on your laptop?


    Also cc'ing @jczogalla @Marco_Boeck in case they have further thoughts...




  • Options
    Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn

    Was there any additional information about this issue after the original post? My own suspicion is that when opening the folder, RM is trying to access all the contents' metadata in the background, which might actually be quite large if the underlying repositories are large.  Whereas if you don't directly open the folder, RM just leaves it alone and thus doesn't slow anything down.  But I am not a RapidMiner developer so I don't really know what is going on in the background, this is just my hypothesis after observations.

    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
Sign In or Register to comment.