🦉 🎤   RapidMiner Wisdom 2020 - CALL FOR SPEAKERS   🦉 🎤
We are inviting all community members to submit proposals to speak at Wisdom 2020 in Boston.
Whether it's a cool RapidMiner trick or a use case implementation, we want to see what you have.
Form link is below and deadline for submissions is November 15. See you in Boston!
Challenge with RM Server - Running out of memory
I am a newbie and this is my first post in the community. We have got a RM server installation on top of a MS SQL server box. We have a job container with 64GB RAM. I have built some workflows using sample data on studio environment and trying to run those processes after necessary changes in server environment connecting to original SQL data tables. These workflows mainly involve some basic data joins and summarization after application of few domain specific business rules.
When I am trying to run a flow, I am quickly running into the issue of Running Out of Memory. The challenge I have is, even the first part of my flow which involves reading few variables from a 40GB dataset is not getting completed. Due to the nature of data and business knowledge involved, I am not in a position to share the XML flow or log files here.
I have few specific questions for the community:
1. How does RM Server handles memory internally? Will the whole source data file be read and kept in memory while processing?
2. What is the maximum database size at source that can be handled by a 64GB single container?
3. Will you recommend RM server for huge data processing operations (i.e. data running closer to a TB in size).