The Altair Community is migrating to a new platform to provide a better experience for you. The RapidMiner Community will merge with the Altair Community at the same time. In preparation for the migration, both communities are on read-only mode from July 15th - July 24th, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here.

"Rapidminer random crashes"

hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
edited June 2019 in Help

I am having some severe problems with random crashes. I added a loop parameters operator to incrementally increase the population size of an optimize selection (evolutionary,parallel) operator. Nothing extreme, just nine increments from 5-50 and I am using six threads. There is 50% available memory while running the process so memory is not the issue.


The problem is that sometimes it works fine and then other times it crashes with an error. I don't get an error if I run the process without looping parameters.  Here is a snapshot of the exception if that helps:-


Kind regards,




Exception: java.util.ConcurrentModificationException
Message: null
Stack trace:



  • Options
    landland RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 2,531 Unicorn



    i know these errors. Best to make a feature request to RapidMiner that the data sets become thread safe :)


    To avoid that you usually can sacrifice some memory and time by inserting Materialize Data operator as first operator in the subprocess. If that does not work: I would rather paralelize the outer loop and use the standard, serial optimize selection operator. Unfortunately for that you would need our Jackhammer extension as you otherwise don't have a parallel loop. See below link for more details.




  • Options
    hughesfleming68hughesfleming68 Member Posts: 323 Unicorn

    Hi Sebastian,


    I tried many things to solve the problem. I finally took out the loop parameters operater and setup a subprocess for each parameter. In my case that meant creating 10 sub processes which was not too bad. I have run the new process many times now without any crashes using the optimize selection evolutionary parallel without any issues. 


    It seems that combining a loop parameters operater with a threaded operator inside may work in some cases and crash in others. I will certainly make a feature request on this issue as it should work better.


    Thank you for making me aware of your Jackhammer extension. I will take a look. 


    Kind regards,



  • Options
    hughesfleming68hughesfleming68 Member Posts: 323 Unicorn

    I finally managed to get my process stable. Even separating out the loop parameters operator and creating individual subprocesses did not solve the problem completely. I was still getting random crashes. Random in the sense that it would run three or four times and then crash on the fifth.


    Putting a Materialize Data operator inside the optimize parameters evolutionary seems to solve the problem. The side effect is greater cpu usage. I had to drop from six threads to four. Even with four threads I was hitting > 80% CPU usage. Processing times have gone up significantly (about 2x)


    To be honest, this fix seems more like a hack than anything else but I have run out of ideas. The parallel processing extention seems very hit and miss. I could easily justify adding CPU's to this problem but I am concerned about scalability. Is anyone running processes on workstations with with greater than 20 threads? Are there any other issues to consider?


    Kind regards,



  • Options
    MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,525 RM Data Scientist

    Dear Alex,


    any chance you can sent me the process and maybe even the data via pm or mail? I am happy to forward this to our development to have a look on it. Of course i cannot promise anything.



    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Options
    Marco_BoeckMarco_Boeck Administrator, Moderator, Employee, Member, University Professor Posts: 1,996 RM Engineering



    I can tell you that we are already aware of this particular issue and looking into it.




  • Options
    hughesfleming68hughesfleming68 Member Posts: 323 Unicorn

    Thanks Marco and Martin,


    I will note any errors if they pop up and send them in if that will be helpful.





Sign In or Register to comment.