Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.
Valdiation of the model and adjusted R Squared
Learner I
Hey Community,
I have a question regarding the validation of my model (I used the cross validation operator).
I created a prediction model (label: numeric) and therfore used the algorithms "Linear Regression, "Neural Net" and "Deep Learning". For validation I chose the RMSE, the relative error and the squared correlation (R squared).
I read that the R squared gets better as more attributes are chosen. To prevent this, I read that the adjusted R squared should be chosen. Is this possible with RapidMiner Studio or is this already the adjusted R squared?
To improve my model I also tested with the "Select attributes" operator and noticed the following:
When I selected all attributes I had this performance:
Case 1
Linear Regression (RMSE 0,8 I Relative Error 14,2 I R squared 0,63)
Neural Net (RMSE 0,86 I RF 15,91 I R squared 0,65)
Deep Learning (RMSE 0,78 I RF 11,72 I R squared 0,68)
In this case the Deep Learning should be the best model.
Now I removed some attributes for modeling and got the following results:
Case 2
Linear Regression (RMSE 0,79 I Relative Error 14,0 I R squared 0,68)
Neural Net (RMSE 0,89 I RF 17,29 I R squared 0,66)
Deep Learning (RMSE 0,79 I RF 13,19 I R squared 0,65)
I do not really know if the Linear Regression in second case is the better model as the Deep Learning in first case (R squared got better but Relative Error got worse). Can somebody help?
Thanks a lot!
I have a question regarding the validation of my model (I used the cross validation operator).
I created a prediction model (label: numeric) and therfore used the algorithms "Linear Regression, "Neural Net" and "Deep Learning". For validation I chose the RMSE, the relative error and the squared correlation (R squared).
I read that the R squared gets better as more attributes are chosen. To prevent this, I read that the adjusted R squared should be chosen. Is this possible with RapidMiner Studio or is this already the adjusted R squared?
To improve my model I also tested with the "Select attributes" operator and noticed the following:
When I selected all attributes I had this performance:
Case 1
Linear Regression (RMSE 0,8 I Relative Error 14,2 I R squared 0,63)
Neural Net (RMSE 0,86 I RF 15,91 I R squared 0,65)
Deep Learning (RMSE 0,78 I RF 11,72 I R squared 0,68)
In this case the Deep Learning should be the best model.
Now I removed some attributes for modeling and got the following results:
Case 2
Linear Regression (RMSE 0,79 I Relative Error 14,0 I R squared 0,68)
Neural Net (RMSE 0,89 I RF 17,29 I R squared 0,66)
Deep Learning (RMSE 0,79 I RF 13,19 I R squared 0,65)
I do not really know if the Linear Regression in second case is the better model as the Deep Learning in first case (R squared got better but Relative Error got worse). Can somebody help?
Thanks a lot!
Tagged:
0
Answers
As far as what model is best, there is no simple way to answer this question based purely on these performance metrics. You have to know the use case to understand the tradeoffs between a slightly higher relative error vs Rsq. It would also probably help to look at the underlying data to determine whether the relationship does look like it should be linear or not. Additional feature engineering may be helpful in improving all of the model fits.
Lindon Ventures
Data Science Consulting from Certified RapidMiner Experts