Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

After completing a process, does RM release memory it consumed during execution?

tennenrishintennenrishin Member Posts: 177 Contributor II
edited November 2018 in Help
I have a process that, under certain conditions, consumes about 14GB of memory during its execution. This process does not have any outputs, and therefore no RM results are generated. It only generates reports as files.

Isn't RM supposed to release this memory after the process has completed?

Answers

  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Not necessarily, it is up to the java garbage collection. I don't know how much you are familiar with the guts of java, but even if RapidMiner declares memory as free, it does not necessarily mean that java frees it up for system use directly.
    However, to trigger the garbage collection manually and free all memory that RapidMiner does not need anymore, you can use the Free Memory operator.

    Additionally, RapidMiner stores intermediate results at each output of each operator, which may also consume some memory.


    Best regards,
    Marius
  • tennenrishintennenrishin Member Posts: 177 Contributor II
    Thanks Marius. More technically, I should perhaps have asked: Does RM retain references to objects in that memory (thus preventing their collection by the GC)?

    Because to all appearances, it does. I have experimented quite a bit in the past (including trying the Free Memory operator), but exiting and restarting RM seems to be the only way to get things back to normal after running (and completing/stopping) this memory-intensive process.

    Typically, I have to run a process or two, restart RM, run a process or two, restart RM, run a process or two, etc. If I just keep running processes without restarting RM, then RM grinds to a halt with an error message about insufficient memory.

    (I can't recall why, but my suspicion is that it may be something in the reporting extension holding onto objects after the report-generating process has long completed. Is such a thing possible, or is there some internal policy through which RM ensures that no references of resources used by operators survive?)

    Any advice/info would be appreciated.
    Thanks in advance
    Isak
  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    It seems that the Reporting Extension does not free all memory. I don't know yet the exact reason why, but I already created an internal issue for that.
  • tennenrishintennenrishin Member Posts: 177 Contributor II
    Thanks Marius!
Sign In or Register to comment.