Options
"Weight by user spec/ scale by weights"
Contributor II
Hello guys
I have been using "weight by user specification" and "scale by weights" to transform my data by weighting the attributes before it enters the learner model (default model,SVM or neural Net).
The data seems to be transformed in the correct way by being multiplied by the specified weights.
However, the problem comes with the prediction; the prediction of the label attribute (and associated probability) is exactly the same using the original, non-weighted data as it is using the new weighted data.
How can this be possible? How can the prediction be exactly the same for 2 different types of data? Surely weighting the attributes in different ways will affect the data and give a different prediction to the situation where the attributes were all weighted the same?
I have tried this approach of using weighted and unweighted attributes on the same data for many different learner models (eg SVM, NN, default model) and it doesnt seem to make a difference
Maybe I am not understanding the models correctly....but as I see it transforming the input data via weights should change the prediction.
Please see my process below; I have used the "gold data" from neural market trends to try and predict the "up or down trend"
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.1.002">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="5.1.002" expanded="true" name="Process">
<process expanded="true" height="296" width="882">
<operator activated="true" class="retrieve" compatibility="5.1.002" expanded="true" height="60" name="Retrieve" width="90" x="26" y="60">
<parameter key="repository_entry" value="../Gold/label"/>
</operator>
<operator activated="true" class="weight_by_user_specification" compatibility="5.1.002" expanded="true" height="76" name="Weight by User Specification (2)" width="90" x="159" y="46">
<parameter key="normalize_weights" value="false"/>
<list key="name_regex_to_weights">
<parameter key="^GDAXI" value="100.0"/>
<parameter key="CD" value="100.0"/>
</list>
</operator>
<operator activated="true" class="scale_by_weights" compatibility="5.1.002" expanded="true" height="76" name="Scale by Weights" width="90" x="296" y="51"/>
<operator activated="true" class="default_model" compatibility="5.1.002" expanded="true" height="76" name="Default Model" width="90" x="514" y="75"/>
<operator activated="true" class="retrieve" compatibility="5.1.002" expanded="true" height="60" name="Retrieve (2)" width="90" x="51" y="187">
<parameter key="repository_entry" value="../Gold/no label"/>
</operator>
<operator activated="true" class="weight_by_user_specification" compatibility="5.1.002" expanded="true" height="76" name="Weight by User Specification" width="90" x="246" y="165">
<parameter key="normalize_weights" value="false"/>
<list key="name_regex_to_weights">
<parameter key="^GDAXI" value="100.0"/>
<parameter key="CD" value="100.0"/>
</list>
</operator>
<operator activated="true" class="scale_by_weights" compatibility="5.1.002" expanded="true" height="76" name="Scale by Weights (2)" width="90" x="380" y="165"/>
<operator activated="true" class="apply_model" compatibility="5.1.002" expanded="true" height="76" name="Apply Model" width="90" x="779" y="155">
<list key="application_parameters"/>
</operator>
<connect from_op="Retrieve" from_port="output" to_op="Weight by User Specification (2)" to_port="example set"/>
<connect from_op="Weight by User Specification (2)" from_port="weights" to_op="Scale by Weights" to_port="weights"/>
<connect from_op="Weight by User Specification (2)" from_port="example set" to_op="Scale by Weights" to_port="example set"/>
<connect from_op="Scale by Weights" from_port="example set" to_op="Default Model" to_port="training set"/>
<connect from_op="Default Model" from_port="model" to_op="Apply Model" to_port="model"/>
<connect from_op="Retrieve (2)" from_port="output" to_op="Weight by User Specification" to_port="example set"/>
<connect from_op="Weight by User Specification" from_port="weights" to_op="Scale by Weights (2)" to_port="weights"/>
<connect from_op="Weight by User Specification" from_port="example set" to_op="Scale by Weights (2)" to_port="example set"/>
<connect from_op="Scale by Weights (2)" from_port="example set" to_op="Apply Model" to_port="unlabelled data"/>
<connect from_op="Apply Model" from_port="labelled data" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>
Sorry for such a long post, but I have been at this for hours and would really appreciate some help
Cheers
Schills
I have been using "weight by user specification" and "scale by weights" to transform my data by weighting the attributes before it enters the learner model (default model,SVM or neural Net).
The data seems to be transformed in the correct way by being multiplied by the specified weights.
However, the problem comes with the prediction; the prediction of the label attribute (and associated probability) is exactly the same using the original, non-weighted data as it is using the new weighted data.
How can this be possible? How can the prediction be exactly the same for 2 different types of data? Surely weighting the attributes in different ways will affect the data and give a different prediction to the situation where the attributes were all weighted the same?
I have tried this approach of using weighted and unweighted attributes on the same data for many different learner models (eg SVM, NN, default model) and it doesnt seem to make a difference
Maybe I am not understanding the models correctly....but as I see it transforming the input data via weights should change the prediction.
Please see my process below; I have used the "gold data" from neural market trends to try and predict the "up or down trend"
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.1.002">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="5.1.002" expanded="true" name="Process">
<process expanded="true" height="296" width="882">
<operator activated="true" class="retrieve" compatibility="5.1.002" expanded="true" height="60" name="Retrieve" width="90" x="26" y="60">
<parameter key="repository_entry" value="../Gold/label"/>
</operator>
<operator activated="true" class="weight_by_user_specification" compatibility="5.1.002" expanded="true" height="76" name="Weight by User Specification (2)" width="90" x="159" y="46">
<parameter key="normalize_weights" value="false"/>
<list key="name_regex_to_weights">
<parameter key="^GDAXI" value="100.0"/>
<parameter key="CD" value="100.0"/>
</list>
</operator>
<operator activated="true" class="scale_by_weights" compatibility="5.1.002" expanded="true" height="76" name="Scale by Weights" width="90" x="296" y="51"/>
<operator activated="true" class="default_model" compatibility="5.1.002" expanded="true" height="76" name="Default Model" width="90" x="514" y="75"/>
<operator activated="true" class="retrieve" compatibility="5.1.002" expanded="true" height="60" name="Retrieve (2)" width="90" x="51" y="187">
<parameter key="repository_entry" value="../Gold/no label"/>
</operator>
<operator activated="true" class="weight_by_user_specification" compatibility="5.1.002" expanded="true" height="76" name="Weight by User Specification" width="90" x="246" y="165">
<parameter key="normalize_weights" value="false"/>
<list key="name_regex_to_weights">
<parameter key="^GDAXI" value="100.0"/>
<parameter key="CD" value="100.0"/>
</list>
</operator>
<operator activated="true" class="scale_by_weights" compatibility="5.1.002" expanded="true" height="76" name="Scale by Weights (2)" width="90" x="380" y="165"/>
<operator activated="true" class="apply_model" compatibility="5.1.002" expanded="true" height="76" name="Apply Model" width="90" x="779" y="155">
<list key="application_parameters"/>
</operator>
<connect from_op="Retrieve" from_port="output" to_op="Weight by User Specification (2)" to_port="example set"/>
<connect from_op="Weight by User Specification (2)" from_port="weights" to_op="Scale by Weights" to_port="weights"/>
<connect from_op="Weight by User Specification (2)" from_port="example set" to_op="Scale by Weights" to_port="example set"/>
<connect from_op="Scale by Weights" from_port="example set" to_op="Default Model" to_port="training set"/>
<connect from_op="Default Model" from_port="model" to_op="Apply Model" to_port="model"/>
<connect from_op="Retrieve (2)" from_port="output" to_op="Weight by User Specification" to_port="example set"/>
<connect from_op="Weight by User Specification" from_port="weights" to_op="Scale by Weights (2)" to_port="weights"/>
<connect from_op="Weight by User Specification" from_port="example set" to_op="Scale by Weights (2)" to_port="example set"/>
<connect from_op="Scale by Weights (2)" from_port="example set" to_op="Apply Model" to_port="unlabelled data"/>
<connect from_op="Apply Model" from_port="labelled data" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>
Sorry for such a long post, but I have been at this for hours and would really appreciate some help
Cheers
Schills
Tagged:
0
Answers
I can see two problems.
Firstly, the "weight by user specification" fails to create a weight for the attribute ^GDAXI because the ^ is interpreted as some sort of regular experession so it fails to match the attribute. My workaround is to rename the attributes.
Secondly, the default model operator is set to "median" which means it always chooses the median of the class labels which looks like it is "DOWN". Hence applying the model to unlabelled data will always yield "DOWN" regardless of the atttributes. If you change this to something like a neural network and add an XValidation block to output a performance, you can get some sort of view about how well the model would behave on unseen data.
regards
Andrew
Thanks so much for taking the time to reply to my post! It was very informative and solved most my problems!
I have one further question regarding using the Support Vector Machine (SVM);
I have used your method (code) below, but replaced the Neural Net with a SVM as I believe it is more accurate for what I am trying to model (eg FOREX). However, using the SVM gives me the same problems as I mentioned before, in that the predictions and probabilities are exactly the same using weights and then not using weights....do you know why this is, or how to fix?
Is it because the SVM applies its own weights and thus any weights you apply to the data before this is irrelevant?
Is there a way to use your own weights with a SVM, or if this is not possible, is there a similar model to a SVM that you can apply weights to?
Your method below works fine using Neural net and X-validation, but does not work when replacing the modelling operator with a SVM.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.1.002">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="5.1.002" expanded="true" name="Process">
<process expanded="true" height="296" width="882">
<operator activated="true" class="retrieve" compatibility="5.1.002" expanded="true" height="60" name="Retrieve" width="90" x="26" y="60">
<parameter key="repository_entry" value="../Gold/label"/>
</operator>
<operator activated="true" class="support_vector_machine" compatibility="5.1.002" expanded="true" height="112" name="SVM" width="90" x="581" y="75"/>
<operator activated="true" class="retrieve" compatibility="5.1.002" expanded="true" height="60" name="Retrieve (2)" width="90" x="51" y="187">
<parameter key="repository_entry" value="../Gold/no label"/>
</operator>
<operator activated="true" class="apply_model" compatibility="5.1.002" expanded="true" height="76" name="Apply Model" width="90" x="779" y="155">
<list key="application_parameters"/>
</operator>
<connect from_op="Retrieve" from_port="output" to_op="SVM" to_port="training set"/>
<connect from_op="SVM" from_port="model" to_op="Apply Model" to_port="model"/>
<connect from_op="Retrieve (2)" from_port="output" to_op="Apply Model" to_port="unlabelled data"/>
<connect from_op="Apply Model" from_port="labelled data" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>
Compare the output prediction generated with the above code (unweighted) vs your code below (but changing NN to SVM).
Any extra info on this issue would be much appreciated
Cheers
Schills
In short, I don't know the details without spending more time but normalising is important for SVM. I can't be sure but I think all the work to scale attributes would be undone by an automatic normalising that might happen in the operator - see the following link.
http://rapid-i.com/rapidforum/index.php/topic,83.0.html
Generally, the literature often states the benefits of SVMs as being better for highly dimensioned data so it doesn't really matter about the number of attributes; it should ignore irrelevant ones.
SVMs require care to get the parameters right, the link above itself has a link to a guide written by the authors of the libsvm code. There is no SilVer Magic bullet I'm afraid.
regards,
Andrew
I'm interested in Forex so could you expand on why you believe that?
Many thanks.
Thanks again for all the info. I now understand SVM a lot better!
Haddock
I think SVM are best for Forex and other financial predictions because it works well for a large number of attributes. It is a great starting point and gives you a real good overview. Comparing the predictions to historical values, the SVM appeared to be the most accurate