"Time series filtering / intervals"

AlexusAlexus Member Posts: 7 Contributor II
edited June 2019 in Help
Hey there,

I would like to know if I can prepare time series (financial, i.e. eurusd) in the following ways in rapidminer:

1. from all data, i.e. extract all mondays and make a new table
2. from all data, i.e. extract only 8:12am mondays and make a new table
3. given a table with dates and times, for example 10 datasets that contain different dates and times, rapidminer should then look at another table with finanical (tick) data and make a new table which contains only data starting with the dates from the first table, plus X hours afterwards.

and similar tasks. In short, I would like to make new time series from (tick)data with the parameters I choose, and/or from dates from other tables.

Is that possible with rapidminer?

And another question: Whats the technical term for such actions?

Answers

  • AlexusAlexus Member Posts: 7 Contributor II
    And if so, which version of rapid miner is capable of doing that? Is the community edition enough?
  • wesselwessel Member Posts: 537 Maven
    Hey,

    With high probability, yes to all questions.
    Here is a demo.

    Best regards,

    Wessel



    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.2.006">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.2.006" expanded="true" name="Process">
        <process expanded="true" height="392" width="165">
          <operator activated="true" class="generate_sales_data" compatibility="5.2.006" expanded="true" height="60" name="Generate Sales Data" width="90" x="45" y="30"/>
          <operator activated="true" class="date_to_numerical" compatibility="5.2.006" expanded="true" height="76" name="Date to Numerical" width="90" x="200" y="29">
            <parameter key="attribute_name" value="date"/>
            <parameter key="time_unit" value="day"/>
            <parameter key="day_relative_to" value="week"/>
            <parameter key="keep_old_attribute" value="true"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="5.2.006" expanded="true" height="76" name="Filter Examples" width="90" x="324" y="27">
            <parameter key="condition_class" value="attribute_value_filter"/>
            <parameter key="parameter_string" value="date_day=1"/>
          </operator>
          <connect from_op="Generate Sales Data" from_port="output" to_op="Date to Numerical" to_port="example set input"/>
          <connect from_op="Date to Numerical" from_port="example set output" to_op="Filter Examples" to_port="example set input"/>
          <connect from_op="Filter Examples" from_port="example set output" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
  • AlexusAlexus Member Posts: 7 Contributor II
    Hey, thats great! You have NO idea how long I have been searching for software thats capable of that, unfortunately I did not know what to search for exactly...

    Is there a tutorial I can look at, that you know, to get me started?
  • AlexusAlexus Member Posts: 7 Contributor II
    And whats the technical term for this kind of operations that I want to do?
  • wesselwessel Member Posts: 537 Maven
    There is no technical term that I am aware off.
    The best definition I can come up with is:
    http://en.wikipedia.org/wiki/Unix_time

    All these operations boil down to the basics of converting a date back to an integer holding the number of milliseconds after 1970.
    E.g. http://www.vogella.com/articles/JavaDateTimeAPI/article.html

    Best regards,

    Wessel
  • AlexusAlexus Member Posts: 7 Contributor II
    I don't quite get it - how does i.e. extracting all mondays out of a time series that contains monday to friday into a new time series that only contains mondays relate to unix time?
  • AlexusAlexus Member Posts: 7 Contributor II
    I don't know if we understand each other? in MYSQL terms I would try a command like (not a working example I guess):

    select * from table where date = monday and time between 8-12

    or something along those lines. I have not found out yet how to do similar things in rapidminer
  • wesselwessel Member Posts: 537 Maven
    Alexus wrote:

    I don't quite get it - how does i.e. extracting all mondays out of a time series that contains monday to friday into a new time series that only contains mondays relate to unix time?
    To make this calculations, Rapid MIner uses code that is provided in any standard Java Installation.
    E.g. http://docs.oracle.com/javase/1.4.2/docs/api/java/util/Calendar.html

    Think about it, how would you implement this code if you had to do it from scratch?
    For a lot of stuff milliseconds are really useful.
    If you represent time as an integer holding the number of milliseconds after 1970 you can simple compare dates using the ">" operator.
    And you can do stuff like:
    if A = Monday
    then so is A+=7 *24 * 60 * 60 * 10000
  • AlexusAlexus Member Posts: 7 Contributor II
    I see. Thanks for your explanation. I conclude from your examples that its not yet possible to to this kind of task via the gui, right? If I have to do this via coding(i.e. in java), then I think preparing the data via SQL might be the easier way.
  • wesselwessel Member Posts: 537 Maven
    I do not agree with your conclusion.
    Did my first example not show you how to do this within Rapid Miner?
    And then I tried to explain you the underlying methods that make it work.
Sign In or Register to comment.