Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

Problems with the RAM

SimonKSimonK Member Posts: 20 Learner II
in Help
Hi,
I work with 23 attributes with about 6.5 million values each. When using different operators, the memory of my laptop is often not enough.
Is there an operator to remove rows in which the values hardly change or to reduce the size of the data set?

Best Answer

  • BalazsBaranyBalazsBarany Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert Posts: 955 Unicorn
    Solution Accepted
    Hi SimonK,

    for attributes that don't change a lot you would use "Remove useless attributes". This drops attributes that have little variation in them.

    For rows you need a way to define what you mean by "hardly change" if you want to systematically remove them. But then you would probably change the data distribution.

    If you have so much data, sampling is a good way to reduce the data size. You could try different sampling sizes. At some point, models don't change anymore even when throwing more data at them.

    Regards,
    Balázs

Answers

Sign In or Register to comment.