Conditional Macros

josh321josh321 Member Posts: 9 Contributor II
edited November 2018 in Help
I'm having a bit of trouble with macros and hoping to get some ideas. Currently, I have a process that looks like the image below. The loop collection uses the macro values to calculate attribute values which are added to the data set. The problem I'm having is that the macro values seem to change dependant on processing order and not be dependant on the throughput at all. I can eliminate the collection process, order the process execution to go tree by tree and simply have the final processes duplicated for each case in the tree. I thought this rather cumbersome and I'm hoping for a more elegant solution. Any suggestions?

Thanks,
Josh

image

Answers

  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Hi,

    as always, actual process xml is probably more helpful. Currently I don't understand completely what you are trying to do; however concerning the macros it should be noted, that they are *global* variables, i.e. if they change in one branch or subprocess or anywhere else in the process, they also change at all other places.

    Best, Marius
  • josh321josh321 Member Posts: 9 Contributor II
    Maybe macro's aren't the way to go then.  Here's some example code of what I'm trying to do. I'm trying to calculate a value based off a coefficient that changes depending on which filter is applied. Is there another way of storing such a value? I know I could store a value for each example in the data set, but with several hundred thousand examples, thats a lot of unessessary data to store when I only need a handful of values stored.

    Thanks for your help. 

    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.0">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.0.10" expanded="true" name="Process">
        <process expanded="true" height="566" width="1351">
          <operator activated="true" class="read_excel" compatibility="5.0.10" expanded="true" height="60" name="Read Excel" width="90" x="45" y="255">
            <list key="annotations"/>
            <list key="data_set_meta_data_information"/>
          </operator>
          <operator activated="true" class="multiply" compatibility="5.0.10" expanded="true" height="94" name="Multiply" width="90" x="179" y="255"/>
          <operator activated="true" class="filter_examples" compatibility="5.0.10" expanded="true" height="76" name="Filter Examples" width="90" x="313" y="165">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="attribute_1 &gt; 50"/>
          </operator>
          <operator activated="true" class="set_macros" compatibility="5.0.10" expanded="true" height="76" name="Set Macros" width="90" x="447" y="165">
            <list key="macros">
              <parameter key="a1" value="20"/>
              <parameter key="a0" value="13"/>
            </list>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.0.10" expanded="true" height="76" name="Filter Examples (2)" width="90" x="313" y="390">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="attribute_1 &lt;= 50"/>
          </operator>
          <operator activated="true" class="set_macros" compatibility="5.0.10" expanded="true" height="76" name="Set Macros (2)" width="90" x="447" y="390">
            <list key="macros">
              <parameter key="a1" value="34"/>
              <parameter key="a0" value="9.43"/>
            </list>
          </operator>
          <operator activated="true" class="multiply" compatibility="5.0.10" expanded="true" height="94" name="Multiply (2)" width="90" x="581" y="165"/>
          <operator activated="true" class="filter_examples" compatibility="5.0.10" expanded="true" height="76" name="Filter Examples (3)" width="90" x="715" y="120">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="attribute_3 &gt;34.5"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.0.10" expanded="true" height="76" name="Filter Examples (4)" width="90" x="715" y="210">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="attribute_3 &lt;=34.5"/>
          </operator>
          <operator activated="true" class="multiply" compatibility="5.0.10" expanded="true" height="94" name="Multiply (3)" width="90" x="581" y="390"/>
          <operator activated="true" class="filter_examples" compatibility="5.0.10" expanded="true" height="76" name="Filter Examples (6)" width="90" x="715" y="435">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="attribute_3 &lt;=34.5"/>
          </operator>
          <operator activated="true" class="set_macros" compatibility="5.0.10" expanded="true" height="94" name="Set Macros (4)" width="90" x="916" y="390">
            <list key="macros">
              <parameter key="b1" value="66"/>
              <parameter key="b0" value="53"/>
            </list>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.0.10" expanded="true" height="76" name="Filter Examples (5)" width="90" x="715" y="345">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="attribute_3 &gt;34.5"/>
          </operator>
          <operator activated="true" class="set_macros" compatibility="5.0.10" expanded="true" height="94" name="Set Macros (3)" width="90" x="849" y="120">
            <list key="macros">
              <parameter key="b1" value="34.54"/>
              <parameter key="b0" value="52.32"/>
            </list>
          </operator>
          <operator activated="true" class="collect" compatibility="5.0.10" expanded="true" height="130" name="Collect" width="90" x="1050" y="255"/>
          <operator activated="true" class="loop_collection" compatibility="5.0.10" expanded="true" height="76" name="Loop Collection" width="90" x="1184" y="255">
            <process expanded="true" height="656" width="948">
              <operator activated="true" class="generate_attributes" compatibility="5.0.10" expanded="true" height="76" name="Generate Attributes" width="90" x="112" y="30">
                <list key="function_descriptions">
                  <parameter key="calculated_value_1" value="%{a1} * attribute_1 + %{a0}"/>
                  <parameter key="calculated_value_2" value="%{b1} * attribute_2 + %{b0}"/>
                </list>
              </operator>
              <connect from_port="single" to_op="Generate Attributes" to_port="example set input"/>
              <connect from_op="Generate Attributes" from_port="example set output" to_port="output 1"/>
              <portSpacing port="source_single" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Read Excel" from_port="output" to_op="Multiply" to_port="input"/>
          <connect from_op="Multiply" from_port="output 1" to_op="Filter Examples" to_port="example set input"/>
          <connect from_op="Multiply" from_port="output 2" to_op="Filter Examples (2)" to_port="example set input"/>
          <connect from_op="Filter Examples" from_port="example set output" to_op="Set Macros" to_port="through 1"/>
          <connect from_op="Set Macros" from_port="through 1" to_op="Multiply (2)" to_port="input"/>
          <connect from_op="Filter Examples (2)" from_port="example set output" to_op="Set Macros (2)" to_port="through 1"/>
          <connect from_op="Set Macros (2)" from_port="through 1" to_op="Multiply (3)" to_port="input"/>
          <connect from_op="Multiply (2)" from_port="output 1" to_op="Filter Examples (3)" to_port="example set input"/>
          <connect from_op="Multiply (2)" from_port="output 2" to_op="Filter Examples (4)" to_port="example set input"/>
          <connect from_op="Filter Examples (3)" from_port="example set output" to_op="Set Macros (3)" to_port="through 1"/>
          <connect from_op="Filter Examples (4)" from_port="example set output" to_op="Set Macros (4)" to_port="through 1"/>
          <connect from_op="Multiply (3)" from_port="output 1" to_op="Filter Examples (5)" to_port="example set input"/>
          <connect from_op="Multiply (3)" from_port="output 2" to_op="Filter Examples (6)" to_port="example set input"/>
          <connect from_op="Filter Examples (6)" from_port="example set output" to_op="Set Macros (4)" to_port="through 2"/>
          <connect from_op="Set Macros (4)" from_port="through 1" to_op="Collect" to_port="input 3"/>
          <connect from_op="Set Macros (4)" from_port="through 2" to_op="Collect" to_port="input 4"/>
          <connect from_op="Filter Examples (5)" from_port="example set output" to_op="Set Macros (3)" to_port="through 2"/>
          <connect from_op="Set Macros (3)" from_port="through 1" to_op="Collect" to_port="input 1"/>
          <connect from_op="Set Macros (3)" from_port="through 2" to_op="Collect" to_port="input 2"/>
          <connect from_op="Collect" from_port="collection" to_op="Loop Collection" to_port="collection"/>
          <connect from_op="Loop Collection" from_port="output 1" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>


  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Well, I fear storing the coefficients for each example is the way to go...
  • haddockhaddock Member Posts: 849 Maven
    Hi Guys!

    I hate to see a good macro thrown away, especially here in France where we eat anything, as long as it is well cooked and set of by the right wine, so let me try a different recipe! 

    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.2.003">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.2.003" expanded="true" name="Process">
        <process expanded="true" height="615" width="1380">
          <operator activated="true" class="generate_data" compatibility="5.2.003" expanded="true" height="60" name="Generate Data" width="90" x="45" y="120">
            <parameter key="number_examples" value="1000000"/>
            <parameter key="attributes_lower_bound" value="0.0"/>
            <parameter key="attributes_upper_bound" value="100.0"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="att1&gt;50" width="90" x="179" y="120">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att1 &gt; 50"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="att3&gt;34.5" width="90" x="313" y="30">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att3 &gt;34.5"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="!att3&gt;34.5" width="90" x="447" y="120">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att3 &gt;34.5"/>
            <parameter key="invert_filter" value="true"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="!att1&gt;50" width="90" x="313" y="210">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att1 &gt; 50"/>
            <parameter key="invert_filter" value="true"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="att3&gt;34.5 (2)" width="90" x="447" y="255">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att3&gt;34.5"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="!att3&gt;34.5 (2)" width="90" x="581" y="300">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att3&gt;34.5"/>
            <parameter key="invert_filter" value="true"/>
          </operator>
          <operator activated="true" class="collect" compatibility="5.2.003" expanded="true" height="130" name="Collect" width="90" x="715" y="75"/>
          <operator activated="true" class="loop_collection" compatibility="5.2.003" expanded="true" height="76" name="Loop Collection" width="90" x="916" y="75">
            <parameter key="unfold" value="true"/>
            <process expanded="true" height="615" width="888">
              <operator activated="true" class="set_macros" compatibility="5.2.003" expanded="true" height="76" name="Set Macros" width="90" x="220" y="27">
                <list key="macros">
                  <parameter key="a1" value="if(%{a}==1||%{a}==2,20,34)"/>
                  <parameter key="a0" value="if(%{a}==1||%{a}==2,13,9.43)"/>
                  <parameter key="b1" value="if(%{a}==1||%{a}==3,34.54,66)"/>
                  <parameter key="b0" value="if(%{a}==1||%{a}==3,52.32,53)"/>
                </list>
              </operator>
              <operator activated="true" class="generate_attributes" compatibility="5.2.003" expanded="true" height="76" name="Generate Attributes (2)" width="90" x="447" y="30">
                <list key="function_descriptions">
                  <parameter key="calculated_value_1" value="%{a1} * att1 + %{a0}"/>
                  <parameter key="calculated_value_2" value="%{b1} * att2 + %{b0}"/>
                </list>
              </operator>
              <connect from_port="single" to_op="Set Macros" to_port="through 1"/>
              <connect from_op="Set Macros" from_port="through 1" to_op="Generate Attributes (2)" to_port="example set input"/>
              <connect from_op="Generate Attributes (2)" from_port="example set output" to_port="output 1"/>
              <portSpacing port="source_single" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="append" compatibility="5.2.003" expanded="true" height="76" name="Append" width="90" x="1117" y="75"/>
          <connect from_op="Generate Data" from_port="output" to_op="att1&gt;50" to_port="example set input"/>
          <connect from_op="att1&gt;50" from_port="example set output" to_op="att3&gt;34.5" to_port="example set input"/>
          <connect from_op="att1&gt;50" from_port="original" to_op="!att1&gt;50" to_port="example set input"/>
          <connect from_op="att3&gt;34.5" from_port="example set output" to_op="Collect" to_port="input 1"/>
          <connect from_op="att3&gt;34.5" from_port="original" to_op="!att3&gt;34.5" to_port="example set input"/>
          <connect from_op="!att3&gt;34.5" from_port="example set output" to_op="Collect" to_port="input 2"/>
          <connect from_op="!att1&gt;50" from_port="example set output" to_op="att3&gt;34.5 (2)" to_port="example set input"/>
          <connect from_op="att3&gt;34.5 (2)" from_port="example set output" to_op="Collect" to_port="input 3"/>
          <connect from_op="att3&gt;34.5 (2)" from_port="original" to_op="!att3&gt;34.5 (2)" to_port="example set input"/>
          <connect from_op="!att3&gt;34.5 (2)" from_port="example set output" to_op="Collect" to_port="input 4"/>
          <connect from_op="Collect" from_port="collection" to_op="Loop Collection" to_port="collection"/>
          <connect from_op="Loop Collection" from_port="output 1" to_op="Append" to_port="example set 1"/>
          <connect from_op="Append" from_port="merged set" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
    It's quite similar to Josh's original, but it doesn't use copies; it makes four bags into which the examples are apportioned according to their ingredients. On the side of each bag is written the recipe for the answer, so it's good for mass production, like cassoulet. My cooker does a million beans in a couple of seconds.

    Bon apetit!
  • josh321josh321 Member Posts: 9 Contributor II
    Thanks for the input. I like how you structured your filters, haddock. But I still have the problem with macros. So I've gone ahead and done as Marius suggests and stored the coefficients for each example. It certainly not efficient, but it works.

    Thanks again.
  • haddockhaddock Member Posts: 849 Maven
    Mmm, what's your problem with the macros?
  • josh321josh321 Member Posts: 9 Contributor II
    haddock wrote:

    Mmm, what's your problem with the macros?
    I'm using filters to branch a single data set into 32 sub sets. As the each set is filtered further I was hoping to store a coefficient using a macro. Some of these coefficients are shared among data sets as they branch out. The sample I showed illustrates this when four filters pass through two macros. Finally, the sub sets are collected and a process is looped over them that uses the macros to calculate a value. I was hoping for there to be a way to have the macros applied only to the data sets which pass through them. But it appears that is not possible, so I did as Marius recommended and simply got rid of the macros and stored a coefficient for each example in each data set.
  • haddockhaddock Member Posts: 849 Maven
    Hi,
    I was hoping for there to be a way to have the macros applied only to the data sets which pass through them. But it appears that is not possible,
    You were right to hope, it is possible, and that's exactly what my example did; I've expanded my code to show the coefficients as well, just to be clear.

    Good weekend! 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.2.003">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.2.003" expanded="true" name="Process">
        <process expanded="true" height="615" width="1380">
          <operator activated="true" class="generate_data" compatibility="5.2.003" expanded="true" height="60" name="Generate Data" width="90" x="45" y="120">
            <parameter key="number_examples" value="1000000"/>
            <parameter key="attributes_lower_bound" value="0.0"/>
            <parameter key="attributes_upper_bound" value="100.0"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="att1&gt;50" width="90" x="179" y="120">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att1 &gt; 50"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="att3&gt;34.5" width="90" x="313" y="30">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att3 &gt;34.5"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="!att3&gt;34.5" width="90" x="447" y="120">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att3 &gt;34.5"/>
            <parameter key="invert_filter" value="true"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="!att1&gt;50" width="90" x="313" y="210">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att1 &gt; 50"/>
            <parameter key="invert_filter" value="true"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="att3&gt;34.5 (2)" width="90" x="447" y="255">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att3&gt;34.5"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.003" expanded="true" height="76" name="!att3&gt;34.5 (2)" width="90" x="581" y="300">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="att3&gt;34.5"/>
            <parameter key="invert_filter" value="true"/>
          </operator>
          <operator activated="true" class="collect" compatibility="5.2.003" expanded="true" height="130" name="Collect" width="90" x="715" y="75"/>
          <operator activated="true" class="loop_collection" compatibility="5.2.003" expanded="true" height="76" name="Loop Collection" width="90" x="916" y="75">
            <parameter key="unfold" value="true"/>
            <process expanded="true" height="615" width="888">
              <operator activated="true" class="set_macros" compatibility="5.2.003" expanded="true" height="76" name="Set Macros" width="90" x="220" y="27">
                <list key="macros">
                  <parameter key="a1" value="if(%{a}==1||%{a}==2,20,34)"/>
                  <parameter key="a0" value="if(%{a}==1||%{a}==2,13,9.43)"/>
                  <parameter key="b1" value="if(%{a}==1||%{a}==3,34.54,66)"/>
                  <parameter key="b0" value="if(%{a}==1||%{a}==3,52.32,53)"/>
                </list>
              </operator>
              <operator activated="true" class="generate_attributes" compatibility="5.2.003" expanded="true" height="76" name="Generate Attributes (2)" width="90" x="447" y="30">
                <list key="function_descriptions">
                  <parameter key="calculated_value_1" value="%{a1} * att1 + %{a0}"/>
                  <parameter key="calculated_value_2" value="%{b1} * att2 + %{b0}"/>
                  <parameter key="a0" value="%{a0}"/>
                  <parameter key="a1" value="%{a1}"/>
                  <parameter key="b0" value="%{b0}"/>
                  <parameter key="b1" value="%{b1}"/>
                </list>
              </operator>
              <connect from_port="single" to_op="Set Macros" to_port="through 1"/>
              <connect from_op="Set Macros" from_port="through 1" to_op="Generate Attributes (2)" to_port="example set input"/>
              <connect from_op="Generate Attributes (2)" from_port="example set output" to_port="output 1"/>
              <portSpacing port="source_single" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="append" compatibility="5.2.003" expanded="true" height="76" name="Append" width="90" x="1117" y="75"/>
          <connect from_op="Generate Data" from_port="output" to_op="att1&gt;50" to_port="example set input"/>
          <connect from_op="att1&gt;50" from_port="example set output" to_op="att3&gt;34.5" to_port="example set input"/>
          <connect from_op="att1&gt;50" from_port="original" to_op="!att1&gt;50" to_port="example set input"/>
          <connect from_op="att3&gt;34.5" from_port="example set output" to_op="Collect" to_port="input 1"/>
          <connect from_op="att3&gt;34.5" from_port="original" to_op="!att3&gt;34.5" to_port="example set input"/>
          <connect from_op="!att3&gt;34.5" from_port="example set output" to_op="Collect" to_port="input 2"/>
          <connect from_op="!att1&gt;50" from_port="example set output" to_op="att3&gt;34.5 (2)" to_port="example set input"/>
          <connect from_op="att3&gt;34.5 (2)" from_port="example set output" to_op="Collect" to_port="input 3"/>
          <connect from_op="att3&gt;34.5 (2)" from_port="original" to_op="!att3&gt;34.5 (2)" to_port="example set input"/>
          <connect from_op="!att3&gt;34.5 (2)" from_port="example set output" to_op="Collect" to_port="input 4"/>
          <connect from_op="Collect" from_port="collection" to_op="Loop Collection" to_port="collection"/>
          <connect from_op="Loop Collection" from_port="output 1" to_op="Append" to_port="example set 1"/>
          <connect from_op="Append" from_port="merged set" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>

Sign In or Register to comment.