two problems: create Threshold and T-test results

inceptorfullinceptorfull Member Posts: 44 Contributor II
edited November 2018 in Help
I am creating two models and compare there accuracy: first the Logistic and second Neural network for binary classification

I didnot configure threshold , so is this should make the results wrong?

I tried to create threshold using create and apply threshold after applying the model and before perfomance to be 0.5 but the results is the same, although I look to NN threshould in the results it say threshold is 3.24 so why it is not 0.5??



Second Question I made T-test for the accuracy of both Models with the same data inputs it give me the following results


Probabilities for random values with the same result:
----- 0.892
----- -----
Values smaller than alpha=0.050 indicate a probably significant difference between the mean values!
List of performance values:
0: 0.822 +/- 0.049
1: 0.825 +/- 0.074

so  what :0 mean and 1 mean?? as the table in the form of

A B C

Answers

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,507 RM Data Scientist
    Hi,

    dont get confused by thresholds. In the NN the threshold is something internal (i think something like a constant/bias). The create and apply threshold operators change the decision boundery for confidences. If you set it to 0.7 i needs to have at least a confidence of 0.7 to be classified into a specific class.

    ~Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • inceptorfullinceptorfull Member Posts: 44 Contributor II
    thanks a lot, so when I apply the threshold to 0.5 or I didnit apply it it gives me the same results, is that means threshold is already 0.5? do I need to make threshold for Logisitic?

    also can I export rapidminer results in a form to be input for other programs like spss? to make wilcoxon test?
  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,507 RM Data Scientist
    Hi,

    the default threshold to define whether it is a or b is 0.5, yes.

    You can of course export tables to any other formats (e.g. Write Excel). You can not export models. By the way wilcoxon test is included in the stats extension. See: https://oldworldcomputing.com/products/statistics-extension-for-rapidminer . The extension is commercial but there is a free version with limited number of rows.

    ~Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • inceptorfullinceptorfull Member Posts: 44 Contributor II
    thanks a lot for reply, it works fine but
    is there a way to export the descriptive statics of my data ??
    also I want to know how to export the parameters of Neural network for example in a file ? to put in my research ?
    I found write performance and parametrs but the file extenstion is per? so what to do ?
  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,507 RM Data Scientist
    Stats: Only via Aggregate operator

    Parameters for NN: you can use a log and then write the log, for performance: Try performance to data first. this gives you an example set.

    ~Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • inceptorfullinceptorfull Member Posts: 44 Contributor II
    thanks a lot for reply , I am really gratefull for you

    I have a problem I tried to make t-test for the models accuracy and ANOVA test, it work fine using X-validation , but when using Split validation, it gives me error " degree freedoom -2", donot know why that? would you help me in that?

    although I want to know , when using the t-test for accuracy, what data to export to input it in SPPSS? is the accuracy rate? so I write it to Excel then put to in SPPSS? or what? I am confused so I can do more statistical tests in my output?
  • David_ADavid_A Administrator, Moderator, Employee, RMResearcher, Member Posts: 297 RM Research
    Regarding your error message:

    Could you provide me a copy of your process?
    The error message indicates, that your test set is either to small or to large.
Sign In or Register to comment.