FeatureSelect/RemoveUselessFeatures and Holdout Sets

Answers

• Options

  TobiasMalbrecht Moderator, Employee, Member Posts: 295 RM Product Management

  June 2008

  Hi Mike,
  
  unfortunately I did not quite understand what exactly you want to do and in which order you want to do the steps you mentioned ... Do you want to do a cross-validation on the training set, then a validation on the holdout set? Then why do you want to do the cross-validation at all? Or do you want to incorporate a cross-validation inside the feature selection to determine the best features, learn a model and then test the model on the holdout set? Maybe you can clarify my confusion by posting a sample process XML?
  
  Anyway, addressing the different data structures between the training and the holdout set involving a feature selection during training: every feature selection scheme outputs an AttributeWeights object which holds the information which attributes are selected and which ones are deselected by the feature selection. You may store this AttributeWeights object, load it afterwards and use the AttributeWeightsSelection operator to select the features of the holdout set according to the specification in the AttributeWeights object.
  
  Hope that helps, if not please try to explain your procedure a little bit more detailed or post your process XML.
  Regards,
  Tobias

  0
• Options

  Legacy User Member Posts: 0 Newbie

  June 2008

  Hi Tobias,
  
  Thanks for your help.  My goal of all of this is that I (now) just want to use feature selection, fit a model to my the reduced dataset, apply that model to a holdout set, and write the predictions to a file.  It's not working though. The problem I'm having is that RapidMiner is using the wrong variables when applying a model after featureSelection.  I suspect it's applying by order as opposed to by name?
  
  For example, after feature selection, I have three variables.  If I apply a linear regression model that uses all of the variables, the model applier works.
  
      Var1 Var2 Var3     Intercept
  Value 0.319 0.406 19.104
  Coeff 0.868 -0.824 0.722
  V*C 0.277 -0.335 13.787         -9.924     =  3.805  <-- prediction
  
  However, when I use a learner such as W-SimpleLinearRegression that will produce a model with only 1 variable, my predictions are incorrect. 
  
  For example, the input values are the same, but coeff[3] and the intercept have changed, so I should get the prediction of 6.517.
  
      Var1 Var2 Var3             Intercept
  Value 0.319   0.406 19.104
  Coeff                   0.930
  V*C                   17.767 -11.250 = 6.517 <- correct- prediction
  
  
  but that's not what I'm getting.  It seems that RM is  using var1's value of .319 instead of var3s value of 19.104 when applying the model.
      Var1 Var2 Var3             Intercept
  Value 0.319 0.406 19.104
  Coeff 0.930
  V*C 0.297                           -11.250 -10.953 <-- what I'm getting
  
  So, to summarize.  FeatureSelection seems to be confusing the model applier, and me.  It doesn't seem to be using the right variable when making preditions. 
  
  What am I doing wrong?
  
  Thanks,
  Mike

  0
• Options

  IngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder

  July 2008

  Does this also occurs for the learner LinearRegression, i.e. the non-Weka version?
  
  Cheers,
  Ingo

  0