Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

How to DFT

tennenrishintennenrishin Member Posts: 177 Contributor II
edited November 2018 in Help
I want to DFT a series, discard frequencies above a certain threshold, and then DFT it back again. In other words, I want to pass a (recurrent) series through an optimal low-pass filter.

I am stuck at the first step. I cannot get the full DFT of a series as an exampleset. In the following process, the first and second outputs are equivalent (as expected) but the third output has reset the phase of the sinusoid, presumably because the Series to Data operator outputs only the magnitude and discards phase of each element. How can I get hold of the phase information? (Or alternatively, the real/imaginary component.)

Or alternatively, how can I accomplish my original goal?

Answers

  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    You forgot to attach the process.
    However, there is the Fourier Transformation operator that works directly on an example set. Did you try that one?

    Best regards,
    Marius
  • tennenrishintennenrishin Member Posts: 177 Contributor II
    Sorry about that! Here is the process, but all it does is demonstrate that the phase information is lost during the Series to Data conversion. (Compare the series plots of the first two outputs to that of the last.) 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.3.008">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.3.008" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="generate_data" compatibility="5.3.008" expanded="true" height="60" name="Generate Data" width="90" x="45" y="30"/>
          <operator activated="true" class="generate_id" compatibility="5.3.008" expanded="true" height="76" name="Generate ID" width="90" x="179" y="30"/>
          <operator activated="true" class="generate_attributes" compatibility="5.3.008" expanded="true" height="76" name="Generate Attributes" width="90" x="313" y="30">
            <list key="function_descriptions">
              <parameter key="x" value="sin(2*pi*(id-1)/100)"/>
            </list>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="5.3.008" expanded="true" height="76" name="Select Attributes" width="90" x="447" y="30">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="x"/>
            <parameter key="attributes" value="|t|x"/>
            <parameter key="include_special_attributes" value="true"/>
          </operator>
          <operator activated="true" class="materialize_data" compatibility="5.3.008" expanded="true" height="76" name="Materialize Data" width="90" x="581" y="30"/>
          <operator activated="true" class="materialize_data" compatibility="5.3.008" expanded="true" height="76" name="Materialize Data (2)" width="90" x="45" y="165"/>
          <operator activated="true" class="materialize_data" compatibility="5.3.008" expanded="true" height="76" name="Materialize Data (3)" width="90" x="45" y="300"/>
          <operator activated="true" class="series:data_to_series" compatibility="5.3.000" expanded="true" height="60" name="Data To Series (3)" width="90" x="179" y="300">
            <parameter key="series_attribute" value="x"/>
          </operator>
          <operator activated="true" class="series:discrete_fourier_transformation" compatibility="5.3.000" expanded="true" height="60" name="Discrete Fourier Transformation (2)" width="90" x="313" y="300"/>
          <operator activated="true" class="series:series_to_data" compatibility="5.3.000" expanded="true" height="60" name="Series To Data" width="90" x="447" y="255"/>
          <operator activated="true" class="series:data_to_series" compatibility="5.3.000" expanded="true" height="60" name="Data To Series (2)" width="90" x="447" y="345">
            <parameter key="series_attribute" value="DFT(x)_dim_1"/>
          </operator>
          <operator activated="true" class="series:discrete_fourier_transformation" compatibility="5.3.000" expanded="true" height="60" name="Discrete Fourier Transformation (3)" width="90" x="581" y="300">
            <parameter key="direction" value="frequency --&gt; time"/>
          </operator>
          <operator activated="true" class="series:series_to_data" compatibility="5.3.000" expanded="true" height="60" name="Series To Data (3)" width="90" x="715" y="300"/>
          <operator activated="true" class="series:data_to_series" compatibility="5.3.000" expanded="true" height="60" name="Data To Series" width="90" x="179" y="165">
            <parameter key="series_attribute" value="x"/>
          </operator>
          <operator activated="true" class="series:discrete_fourier_transformation" compatibility="5.3.000" expanded="true" height="60" name="Discrete Fourier Transformation" width="90" x="313" y="165"/>
          <operator activated="true" class="series:discrete_fourier_transformation" compatibility="5.3.000" expanded="true" height="60" name="Discrete Fourier Transformation (4)" width="90" x="581" y="165">
            <parameter key="direction" value="frequency --&gt; time"/>
          </operator>
          <operator activated="true" class="series:series_to_data" compatibility="5.3.000" expanded="true" height="60" name="Series To Data (2)" width="90" x="715" y="165"/>
          <connect from_op="Generate Data" from_port="output" to_op="Generate ID" to_port="example set input"/>
          <connect from_op="Generate ID" from_port="example set output" to_op="Generate Attributes" to_port="example set input"/>
          <connect from_op="Generate Attributes" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
          <connect from_op="Select Attributes" from_port="example set output" to_op="Materialize Data" to_port="example set input"/>
          <connect from_op="Materialize Data" from_port="example set output" to_port="result 1"/>
          <connect from_op="Materialize Data" from_port="original" to_op="Materialize Data (2)" to_port="example set input"/>
          <connect from_op="Materialize Data (2)" from_port="example set output" to_op="Data To Series" to_port="example set"/>
          <connect from_op="Materialize Data (2)" from_port="original" to_op="Materialize Data (3)" to_port="example set input"/>
          <connect from_op="Materialize Data (3)" from_port="example set output" to_op="Data To Series (3)" to_port="example set"/>
          <connect from_op="Data To Series (3)" from_port="series" to_op="Discrete Fourier Transformation (2)" to_port="series"/>
          <connect from_op="Discrete Fourier Transformation (2)" from_port="series" to_op="Series To Data" to_port="series"/>
          <connect from_op="Series To Data" from_port="example set" to_op="Data To Series (2)" to_port="example set"/>
          <connect from_op="Data To Series (2)" from_port="series" to_op="Discrete Fourier Transformation (3)" to_port="series"/>
          <connect from_op="Discrete Fourier Transformation (3)" from_port="series" to_op="Series To Data (3)" to_port="series"/>
          <connect from_op="Series To Data (3)" from_port="example set" to_port="result 3"/>
          <connect from_op="Data To Series" from_port="series" to_op="Discrete Fourier Transformation" to_port="series"/>
          <connect from_op="Discrete Fourier Transformation" from_port="series" to_op="Discrete Fourier Transformation (4)" to_port="series"/>
          <connect from_op="Discrete Fourier Transformation (4)" from_port="series" to_op="Series To Data (2)" to_port="series"/>
          <connect from_op="Series To Data (2)" from_port="example set" to_port="result 2"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="18"/>
          <portSpacing port="sink_result 2" spacing="126"/>
          <portSpacing port="sink_result 3" spacing="90"/>
          <portSpacing port="sink_result 4" spacing="0"/>
        </process>
      </operator>
    </process>
    In the Fourier Transformation operator's documentation, I don't know what to make of:
    "This operator uses the label as a function of each attribute..."
    If I just want the Discrete Fourier Transform of some sequence of numbers, how would I use the operator to obtain it?

    I've tried this: 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.3.008">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.3.008" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="generate_data" compatibility="5.3.008" expanded="true" height="60" name="Generate Data" width="90" x="45" y="30"/>
          <operator activated="true" class="generate_id" compatibility="5.3.008" expanded="true" height="76" name="Generate ID" width="90" x="179" y="30"/>
          <operator activated="true" class="generate_attributes" compatibility="5.3.008" expanded="true" height="76" name="Generate Attributes" width="90" x="313" y="30">
            <list key="function_descriptions">
              <parameter key="t" value="id-1"/>
              <parameter key="x" value="sin(2*pi*t/100)"/>
            </list>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="5.3.008" expanded="true" height="76" name="Select Attributes" width="90" x="447" y="30">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attributes" value="|t|x"/>
            <parameter key="include_special_attributes" value="true"/>
          </operator>
          <operator activated="true" class="set_role" compatibility="5.3.008" expanded="true" height="76" name="Set Role" width="90" x="581" y="30">
            <parameter key="attribute_name" value="x"/>
            <parameter key="target_role" value="label"/>
            <list key="set_additional_roles"/>
          </operator>
          <operator activated="true" class="fourier_transformation" compatibility="5.3.008" expanded="true" height="76" name="Fourier Transformation" width="90" x="715" y="30"/>
          <connect from_op="Generate Data" from_port="output" to_op="Generate ID" to_port="example set input"/>
          <connect from_op="Generate ID" from_port="example set output" to_op="Generate Attributes" to_port="example set input"/>
          <connect from_op="Generate Attributes" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
          <connect from_op="Select Attributes" from_port="example set output" to_op="Set Role" to_port="example set input"/>
          <connect from_op="Set Role" from_port="example set output" to_op="Fourier Transformation" to_port="example set input"/>
          <connect from_op="Fourier Transformation" from_port="example set output" to_port="result 1"/>
          <connect from_op="Fourier Transformation" from_port="original" to_port="result 2"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
          <portSpacing port="sink_result 3" spacing="0"/>
        </process>
      </operator>
    </process>
    but this seems incorrect, as a 100-point function would have a 100-point DFT (rather than a 32-point DFT).
Sign In or Register to comment.