X-Validation and Feature engenering

janvanrijnjanvanrijn Member Posts: 15 Contributor II
edited November 2019 in Help
I want to use some feature engineering steps inside a cross-validation loop. For example, I want to evaluate the combination of a PCA (preprocessing) and k-NN (modeling) algorithm. In order to do so, I use the X-Validation operator. However, as PCA changes the features that are available in the original dataset, I cannot in the testing simply connect the model and test set input to the "Apply Model" operator. I probably also need to do something with the preprocessing method.

I can see PCA has a preprocessing model as output, is that what I should use to overcome thing problem? If yes, how?

Answers

  • RalfKlinkenbergRalfKlinkenberg Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, RMResearcher, Member, Unconfirmed, University Professor Posts: 68 RM Founder
    The "Group Models" operator allows to bundle several models into one combined model. You can bundle your pre-processing model (PCA) and your k-NN model into one model that can be passed from the training subprocess of the cross-validation to the test subprocess.
  • janvanrijnjanvanrijn Member Posts: 15 Contributor II
    Dear Ralf,

    Thanks for your quick reply. Something like I did here?
    [img=http://s9.postimg.org/d36r9xaqj/pca.jpg]

    I still get an error, hinting at the wrong set of attributes (included in the screen. )
  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,503 RM Data Scientist
    hi,

    sadly you need to add a Materialze data infront of your apply model operator. There is a bug in the PCA which is fixed in the next relase.

    Cheers,
    MArtin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • janvanrijnjanvanrijn Member Posts: 15 Contributor II
    Awesome. It now works for me. Thanks both :)

    One more question. Suppose I want to use the operators from the "Feature Extraction" extension, that do not output such model (and as far as I'm concerned, just return a weight vector with feature importance) how would I do that?
  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,503 RM Data Scientist
    Hi,

    i think you simply use select by weights afterwards. Thats it. Usually all models know on which attributes they are applied

    By the way: My personal favorite is MRMR :)

    best,
    martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • janvanrijnjanvanrijn Member Posts: 15 Contributor II
    Seems to work pretty well indeed, thanks :)

    One exception though, k-NN.. Whichever preprocessing method I choose, it always obtains the same scores.
  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,503 RM Data Scientist
    Hi,

    which version of RM are you using?
    There are two ways how a application of the model can be done. Either via the attribute names or the attribute location (first attribute, second attribute.). All models implemented by us should take the col. name. There was something with k-NN, that it consideres something wrong. I know that it is fixed. But if you use 5.3 the bug is of course in.

    In this case you need to get the weight vector to the apply side using the through port. For application outside of the X-Val you need to get the Vector out of the x-val. Remember/recall is the way to go here.

    Regards,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
Sign In or Register to comment.