02-21-2017 08:47 PM - edited 02-21-2017 08:50 PM
Typically, in the absence of knowledge about the relative cost of missclassification errors a classifier shoud classify an observation as a member of the "True Class" if Probability(True) > 0.5. That's the behavior of most classifiers in Rapidminer (including W-Logistic).
The new classifier "Logistic Regression" seems to be the exception. This classifier classifies an observation as True if Prob(True) > 0.3 (or in the Rapidminer terminology : if Confidence(True) > 0.3). I'm attaching a process showings this behavior. Just run it. Plot a histogram of Confidence(True) and color it using the variable Prediction(label).
The pic of the histogram is attached to this message too.
02-22-2017 07:47 AM
In the sample process you attached, you use a deep learning operator inside the CV. Is this correct?
02-22-2017 09:11 AM
No. I used the new LogisticRegression operator. I didn't even use cross-validation.
The problem seems to be the GeneralizedLinearRegression routine. I exchanged operator (GLM for Logistic Regression) with the right settings (family=binomial, etc) and I get the same behavior.
02-22-2017 09:31 AM
I see what you're saying. Hmm, let me investigate.
02-22-2017 11:31 AM
That's very curious. Did you try comparing the results of the Weka version of the logistic regression operator?
02-22-2017 01:59 PM
@yyhuang pointed out to me that it might be related to H2O's f1 optimization of binomal data sets for the GLM algo. http://ethen8181.github.io/machine-learning/h2o/h2o_glm/h2o_glm.html
Will continue to investigate.
02-22-2017 02:04 PM
@Telcontar120 I tested this out using the Weka LR and the old Rapidminer SVM LR algo, both give me a label flip at confidence > 0.5 when using a Generate Data operator set to Random Classification.
I think I'm learning toward the internal F1 measure optimization that H20 is doing behind the scenes for binomal labels, but we're looking into this.
02-23-2017 07:43 AM
I tested that too, the other RapidMiner/Weka operators do operate as they should. Based on the H2O documentation, I think it's the F1 optimzation but will confirm.