Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

About the Normalize operator.

RafaRafa Member Posts: 12 Learner II

I have a simple question about the Normalize operator.

Suppose this

  • I have a dataset (m1 examples, n attributes)
  • I normalize them and I get two things: the normalized data and a normalization model.
  • To the original data I add m2 new examples.
  • To this augmented set of data (m1 + m2 examples) I apply the normalization model obtained previously.

Two things must happen:

  1. The new m2 data have been normalized with respect to a model in which they were not present and consequently they do not correspond exactly to what would have been obtained when considering means and variances of the whole set (old plus new)
  2. The m1 normalized values from the initial examples must be identical in both results.

Is this correct?

 Thanks!!!!!!!!!!


Best Answer

  • BalazsBaranyBalazsBarany Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert Posts: 955 Unicorn
    Solution Accepted
    Hi @Rafa

    if I understand your question correctly, this is what you're doing:

    1. Build a normalization model from m1. This gives you the parameters for mean and standard deviation and transforms m1.
    2. You combine m1 (original) and m2 and instead of creating a new normalization model (which would lead to a new calculation of mean and std.dev.) you apply the existing model built from m1 to the combination. 
    3. You compare identical examples (e. g. based on some ID) between the two normalization results and they are not equal.

    This is what I did in this example process, and they *are* identical. 

    <?xml version="1.0" encoding="UTF-8"?><process version="9.8.001">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="9.8.001" expanded="true" name="Process">
        <parameter key="logverbosity" value="init"/>
        <parameter key="random_seed" value="-1"/>
        <parameter key="send_mail" value="never"/>
        <parameter key="notification_email" value=""/>
        <parameter key="process_duration_for_mail" value="30"/>
        <parameter key="encoding" value="SYSTEM"/>
        <process expanded="true">
          <operator activated="true" class="retrieve" compatibility="9.8.001" expanded="true" height="68" name="Retrieve Iris" width="90" x="112" y="34">
            <parameter key="repository_entry" value="//Samples/data/Iris"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="9.8.001" expanded="true" height="103" name="Filter Examples" width="90" x="246" y="34">
            <parameter key="parameter_expression" value=""/>
            <parameter key="condition_class" value="custom_filters"/>
            <parameter key="invert_filter" value="false"/>
            <list key="filters_list">
              <parameter key="filters_entry_key" value="label.does_not_equal.Iris-virginica"/>
            </list>
            <parameter key="filters_logic_and" value="true"/>
            <parameter key="filters_check_metadata" value="true"/>
          </operator>
          <operator activated="true" class="normalize" compatibility="9.8.001" expanded="true" height="103" name="Normalize M1" width="90" x="447" y="34">
            <parameter key="return_preprocessing_model" value="false"/>
            <parameter key="create_view" value="false"/>
            <parameter key="attribute_filter_type" value="all"/>
            <parameter key="attribute" value=""/>
            <parameter key="attributes" value=""/>
            <parameter key="use_except_expression" value="false"/>
            <parameter key="value_type" value="numeric"/>
            <parameter key="use_value_type_exception" value="false"/>
            <parameter key="except_value_type" value="real"/>
            <parameter key="block_type" value="value_series"/>
            <parameter key="use_block_type_exception" value="false"/>
            <parameter key="except_block_type" value="value_series_end"/>
            <parameter key="invert_selection" value="false"/>
            <parameter key="include_special_attributes" value="false"/>
            <parameter key="method" value="Z-transformation"/>
            <parameter key="min" value="0.0"/>
            <parameter key="max" value="1.0"/>
            <parameter key="allow_negative_values" value="false"/>
          </operator>
          <operator activated="true" class="apply_model" compatibility="9.8.001" expanded="true" height="82" name="Apply normalization on original" width="90" x="514" y="187">
            <list key="application_parameters"/>
            <parameter key="create_view" value="false"/>
          </operator>
          <connect from_op="Retrieve Iris" from_port="output" to_op="Filter Examples" to_port="example set input"/>
          <connect from_op="Filter Examples" from_port="example set output" to_op="Normalize M1" to_port="example set input"/>
          <connect from_op="Filter Examples" from_port="original" to_op="Apply normalization on original" to_port="unlabelled data"/>
          <connect from_op="Normalize M1" from_port="example set output" to_port="result 1"/>
          <connect from_op="Normalize M1" from_port="preprocessing model" to_op="Apply normalization on original" to_port="model"/>
          <connect from_op="Apply normalization on original" from_port="labelled data" to_port="result 3"/>
          <connect from_op="Apply normalization on original" from_port="model" to_port="result 2"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
          <portSpacing port="sink_result 3" spacing="0"/>
          <portSpacing port="sink_result 4" spacing="0"/>
        </process>
      </operator>
    </process>


    Would you like to share your process so I can check if it is built in the same way? The expected result of applying the model built from M1 to the (original, unmodified) M1 examples is to receive the same result.

    Best regards,

    Balázs

Answers

  • RafaRafa Member Posts: 12 Learner II
    Add this:
    Normalize method  =  z-transformation.

    It`s clear, answer to 2 in my first post is NOT.
    So, the new question is, WHY NOT???

    Thanks again!!
    R
Sign In or Register to comment.