Weka model accuracy unknown...or random?

Fred12Fred12 Member Posts: 344   Unicorn
edited December 2018 in Product Feedback - Resolved

hi,

I had previously problems with retrieving Weka models from Repository, but when I disabled several extensions, it worked afterwards...

I now have the problem that some Functional Tree operator models from Weka like W-FT and W-LMT bring me accuracy unknown when applying on test data:

 

Unbenannt.PNG

 

I have the sa me problem with W-Rotation Forest and W-Random Forest brings me just 33% arbitrary random performance...

anyone knows what the problem is?

Tagged:
0
0 votes

Duplicate Β· Last Updated

Duplicate of https://community.rapidminer.com/discussion/45514/bug-report-weka-models-no-longer-work-when-installing-web-table-extraction-extention

Comments

  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,761   Unicorn

    Weka models? Do you mean models trained using Weka algorithms in RapidMiner and then stored in the RapidMiner format?

     

    I usually never have a problem and need unload extensions. What if you use a RapidMiner Random Forest as a test to see if you get any P/R results?

  • Fred12Fred12 Member Posts: 344   Unicorn

    I get the usual good / Appropriate results with any Rapidminer models or operators... Random Forest model in Rapidminer works nice...

    but when I train Weka Random Forest model and then store it in repository, retrieve it and test it again on test data the results are weird...

     

    edit: I Just noticed, there is a "Read Weka Model" operator, but it seems not to work, at least it cannot read models stored from the repository... But I don't know how to else read / retrieve them, as there is no "Store Weka Model" operator... is that the reason the model doesn't work or does it not matter how to retrieve or store the model? Is the reason more the Rapidminer format / framework?

  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,761   Unicorn

    I'm not sure why it's doing this but I can confirm something's not right. I just did a test with W-Random Forest, stored the model and tried to retrieve it.  I get this. 

     

    Let me ping someone in development. 

     

    Weka Model.png

     

     

     

    <?xml version="1.0" encoding="UTF-8"?><process version="7.4.000">
    <context>
    <input/>
    <output/>
    <macros/>
    </context>
    <operator activated="true" class="process" compatibility="7.4.000" expanded="true" name="Process">
    <process expanded="true">
    <operator activated="true" class="retrieve" compatibility="7.4.000" expanded="true" height="68" name="Retrieve Iris" width="90" x="112" y="34">
    <parameter key="repository_entry" value="//Samples/data/Iris"/>
    </operator>
    <operator activated="true" class="concurrency:cross_validation" compatibility="7.4.000" expanded="true" height="145" name="Validation" width="90" x="313" y="34">
    <parameter key="sampling_type" value="stratified sampling"/>
    <process expanded="true">
    <operator activated="true" class="weka:W-RandomForest" compatibility="7.3.000" expanded="true" height="82" name="W-RandomForest" width="90" x="209" y="34"/>
    <connect from_port="training set" to_op="W-RandomForest" to_port="training set"/>
    <connect from_op="W-RandomForest" from_port="model" to_port="model"/>
    <portSpacing port="source_training set" spacing="0"/>
    <portSpacing port="sink_model" spacing="0"/>
    <portSpacing port="sink_through 1" spacing="0"/>
    </process>
    <process expanded="true">
    <operator activated="true" class="apply_model" compatibility="7.4.000" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34">
    <list key="application_parameters"/>
    </operator>
    <operator activated="true" class="performance" compatibility="7.4.000" expanded="true" height="82" name="Performance" width="90" x="179" y="34"/>
    <connect from_port="model" to_op="Apply Model" to_port="model"/>
    <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
    <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
    <connect from_op="Performance" from_port="performance" to_port="performance 1"/>
    <connect from_op="Performance" from_port="example set" to_port="test set results"/>
    <portSpacing port="source_model" spacing="0"/>
    <portSpacing port="source_test set" spacing="0"/>
    <portSpacing port="source_through 1" spacing="0"/>
    <portSpacing port="sink_test set results" spacing="0"/>
    <portSpacing port="sink_performance 1" spacing="0"/>
    <portSpacing port="sink_performance 2" spacing="0"/>
    <description align="left" color="blue" colored="true" height="103" resized="true" width="315" x="38" y="137">The model created in the Training step is applied to the current test set (10 %).&lt;br/&gt;The performance is evaluated and sent to the operator results.</description>
    </process>
    <description align="center" color="transparent" colored="false" width="126">A cross-validation evaluating a decision tree model.</description>
    </operator>
    <operator activated="true" class="store" compatibility="7.4.000" expanded="true" height="68" name="Store" width="90" x="514" y="34">
    <parameter key="repository_entry" value="../data/WekaModel"/>
    </operator>
    <operator activated="true" class="retrieve" compatibility="7.4.000" expanded="true" height="68" name="Retrieve Iris (2)" width="90" x="45" y="289">
    <parameter key="repository_entry" value="//Samples/data/Iris"/>
    </operator>
    <operator activated="true" class="retrieve" compatibility="7.4.000" expanded="true" height="68" name="Retrieve WekaModel" width="90" x="514" y="136">
    <parameter key="repository_entry" value="../data/WekaModel"/>
    </operator>
    <operator activated="true" class="apply_model" compatibility="7.4.000" expanded="true" height="82" name="Apply Model (2)" width="90" x="380" y="289">
    <list key="application_parameters"/>
    </operator>
    <connect from_op="Retrieve Iris" from_port="output" to_op="Validation" to_port="example set"/>
    <connect from_op="Validation" from_port="model" to_op="Store" to_port="input"/>
    <connect from_op="Validation" from_port="performance 1" to_port="result 1"/>
    <connect from_op="Retrieve Iris (2)" from_port="output" to_op="Apply Model (2)" to_port="unlabelled data"/>
    <connect from_op="Retrieve WekaModel" from_port="output" to_op="Apply Model (2)" to_port="model"/>
    <connect from_op="Apply Model (2)" from_port="labelled data" to_port="result 2"/>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_result 1" spacing="0"/>
    <portSpacing port="sink_result 2" spacing="0"/>
    <portSpacing port="sink_result 3" spacing="0"/>
    </process>
    </operator>
    </process>

     

     

     

     

     

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,213   Unicorn

    I can confirm that I have experienced a similar problem with Weka models in recent Studio releases.  I am not sure which version this orginated with, but it is problematic.  I have had a similar problem where even trying to review the Weka model in the "model" view doesn't work, it just results in an endless-spinning-hourglass.  It would be great to have this corrected ASAP.

    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • michal_moskalmichal_moskal Member Posts: 3 Contributor I

    Has this issue been resolved now? I think I can still reproduce this in RapidMiner 8

  • sgenzersgenzer 12Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,383  Community Manager

    Issue still exists and and reported to dev team. I will update as needed.


    SG

Sign In or Register to comment.