Speculative Rounds In Forward Selection

Regular Contributor

Speculative Rounds In Forward Selection

Hello all,

I've got a short question:
What feature/attribue chooses the Forward Selection Operator in a speculative round?

Let's assume we have 5 features/attributes, "without increase" - stopping criterium, maximal number of attributes set to 5 and speculative rounds = 2.

1.Round :
 - att1 = 70 %
 - att2 = 60 %
 - att3 = 50 %
 - att4 = 45 %
 - att5 = 80 %
 => attribute 5 will be chosen

2.Round :
- (att5 + att1) = 60 %
- (att5 + att2) = 70 %
- (att5 + att3) = 80 %
- (att5 + att4) = 90 %
=> attribute 4 will be chosen

3.Round :
 - (att5 + att4 + att1) = 70 %
 - (att5 + att4 + att2) = 60 %
 - (att5 + att4 + att3) = 50 %
=> No increase => speculative round

So here's my question again: what attribute will now be chosen?
And what will happen next?

Sorry if this is a stupid question but I didn't find any answer neither here in this forum nor with google.

Thanx a lot in advance,
Super Contributor

Re: Speculative Rounds In Forward Selection

Hi Sasch,

actually, your third round is *not* a speculative round - the Forward Selection always adds one attribute and looks for increases. In your case, the best result was obtained in round 2. Since round 3 does not deliver a better result, the algorithm stops and delivers the best results found, i.e. att5+att4 from round 2.

If you configure one speculative round, the Forward Selection would try a 4th round, even though no increase could be achieved in round 3.

In any case, the attributes that delivered the best performance are returned.

Best regards,
Super Contributor

Re: Speculative Rounds In Forward Selection

Experiment with this process to get a better feeling for what is happening:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.013">
  <operator activated="true" class="process" compatibility="5.3.013" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="5.3.013" expanded="true" height="60" name="Retrieve Iris" width="90" x="45" y="30">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      <operator activated="true" class="optimize_selection_forward" compatibility="5.3.013" expanded="true" height="94" name="Forward Selection" width="90" x="179" y="30">
        <process expanded="true">
          <operator activated="true" class="x_validation" compatibility="5.3.013" expanded="true" height="112" name="Validation" width="90" x="45" y="30">
            <description>A cross-validation evaluating a decision tree model.</description>
            <process expanded="true">
              <operator activated="true" class="decision_tree" compatibility="5.3.013" expanded="true" height="76" name="Decision Tree" width="90" x="45" y="30"/>
              <connect from_port="training" to_op="Decision Tree" to_port="training set"/>
              <connect from_op="Decision Tree" from_port="model" to_port="model"/>
              <portSpacing port="source_training" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="5.3.013" expanded="true" height="76" name="Apply Model" width="90" x="45" y="30">
                <list key="application_parameters"/>
              <operator activated="true" class="performance" compatibility="5.3.013" expanded="true" height="76" name="Performance" width="90" x="179" y="30"/>
              <connect from_port="model" to_op="Apply Model" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
              <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
              <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_averagable 1" spacing="0"/>
              <portSpacing port="sink_averagable 2" spacing="0"/>
          <operator activated="true" class="log" compatibility="5.3.013" expanded="true" height="76" name="Log" width="90" x="246" y="30">
            <list key="log">
              <parameter key="performance" value="operator.Validation.value.performance"/>
              <parameter key="attributes" value="operator.Forward Selection.value.feature_names"/>
          <connect from_port="example set" to_op="Validation" to_port="training"/>
          <connect from_op="Validation" from_port="averagable 1" to_op="Log" to_port="through 1"/>
          <connect from_op="Log" from_port="through 1" to_port="performance"/>
          <portSpacing port="source_example set" spacing="0"/>
          <portSpacing port="sink_performance" spacing="0"/>
      <connect from_op="Retrieve Iris" from_port="output" to_op="Forward Selection" to_port="example set"/>
      <connect from_op="Forward Selection" from_port="attribute weights" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
Regular Contributor

Re: Speculative Rounds In Forward Selection

Hello Marius,
thanks a lot for your answer.

I understood everything so far but I'm still interested in what attribute the Forward Selection picks in round 3
if I configure one speculative round?
3.Round :
  - (att5 + att4 + att1) = 70 %
  - (att5 + att4 + att2) = 60 %
  - (att5 + att4 + att3) = 50 %
=> No increase => speculative round

Will it be att1 because it has the highest rate?

Do have any links to some literature about Forward Selection with speculative rounds?

Thanks again,