Real-Time Financial Data via Alpha Venture API (alternative to Yahoo Finance API)

sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
edited December 2018 in Knowledge Base

 So I believe I have finally found a decent alternative to the old Yahoo Finance API so you can bring financial market data directly into RapidMiner for data analysis.  A company called Alpha Vantage has developed an API that appears to perform the same functions. 

You can get a free API key on their website, and you can use the process below directly into RapidMiner 7.6:

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.000">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="7.6.000" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="set_macros" compatibility="7.6.000" expanded="true" height="68" name="Set Macros" width="90" x="45" y="238">
<list key="macros">
<parameter key="apiKey" value="your-KEY-here"/>
<parameter key="tickerSymbol" value="AAPL"/>
<parameter key="function" value="TIME_SERIES_DAILY"/>
<parameter key="outputSize" value="compact"/>
</list>
</operator>
<operator activated="true" class="generate_data_user_specification" compatibility="7.6.000" expanded="true" height="68" name="Generate Data by User Specification" width="90" x="179" y="238">
<list key="attribute_values"/>
<list key="set_additional_roles"/>
</operator>
<operator activated="true" class="web:enrich_data_by_webservice" compatibility="7.3.000" expanded="true" height="68" name="Enrich Data by Webservice" width="90" x="313" y="238">
<parameter key="query_type" value="Regular Expression"/>
<list key="string_machting_queries"/>
<list key="regular_expression_queries">
<parameter key="foo" value=".*"/>
</list>
<list key="regular_region_queries"/>
<list key="xpath_queries"/>
<list key="namespaces"/>
<list key="index_queries"/>
<list key="jsonpath_queries"/>
<parameter key="url" value="https://www.alphavantage.co/query?function=%{function}&amp;outputsize=%{outputSize}&amp;symbol=%{tickerSymbol}&amp;apikey=%{apiKey}"/>
<list key="request_properties"/>
</operator>
<operator activated="true" class="subprocess" compatibility="7.6.000" expanded="true" height="82" name="Subprocess" width="90" x="447" y="238">
<process expanded="true">
<operator activated="true" class="text:data_to_documents" compatibility="7.5.000" expanded="true" height="68" name="Data to Documents" width="90" x="45" y="34">
<parameter key="select_attributes_and_weights" value="true"/>
<list key="specify_weights">
<parameter key="foo" value="1.0"/>
</list>
</operator>
<operator activated="true" class="text:combine_documents" compatibility="7.5.000" expanded="true" height="82" name="Combine Documents" width="90" x="179" y="34"/>
<operator activated="true" class="text:json_to_data" compatibility="7.5.000" expanded="true" height="82" name="JSON To Data" width="90" x="313" y="34"/>
<operator activated="true" class="numerical_to_real" compatibility="7.6.000" expanded="true" height="82" name="Numerical to Real" width="90" x="447" y="136">
<parameter key="attribute_filter_type" value="regular_expression"/>
<parameter key="regular_expression" value="Time.*"/>
</operator>
<operator activated="true" class="de_pivot" compatibility="7.6.000" expanded="true" height="82" name="De-Pivot" width="90" x="514" y="34">
<list key="attribute_name">
<parameter key="value" value="Time.*"/>
</list>
<parameter key="index_attribute" value="date"/>
<parameter key="create_nominal_index" value="true"/>
</operator>
<operator activated="true" class="split" compatibility="7.6.000" expanded="true" height="82" name="Split" width="90" x="648" y="34">
<parameter key="attribute_filter_type" value="single"/>
<parameter key="attribute" value="date"/>
<parameter key="split_pattern" value="[.]\s"/>
</operator>
<operator activated="true" class="replace" compatibility="7.6.000" expanded="true" height="82" name="Replace" width="90" x="782" y="34">
<parameter key="attribute_filter_type" value="single"/>
<parameter key="attribute" value="date_1"/>
<parameter key="replace_what" value="Time Series \(Daily\)[.]"/>
</operator>
<operator activated="true" class="replace" compatibility="7.6.000" expanded="true" height="82" name="Replace (2)" width="90" x="916" y="34">
<parameter key="attribute_filter_type" value="single"/>
<parameter key="attribute" value="date_1"/>
<parameter key="replace_what" value="[.][0-9]"/>
</operator>
<operator activated="true" class="pivot" compatibility="7.6.000" expanded="true" height="82" name="Pivot" width="90" x="1050" y="34">
<parameter key="group_attribute" value="date_1"/>
<parameter key="index_attribute" value="date_2"/>
<parameter key="consider_weights" value="false"/>
</operator>
<operator activated="true" class="rename_by_replacing" compatibility="7.6.000" expanded="true" height="82" name="Rename by Replacing" width="90" x="1184" y="34">
<parameter key="replace_what" value="value[_]"/>
</operator>
<operator activated="true" class="rename_by_replacing" compatibility="7.6.000" expanded="true" height="82" name="Rename by Replacing (2)" width="90" x="1318" y="34">
<parameter key="replace_what" value="Meta Data[.][0-9][.]\s"/>
</operator>
<operator activated="true" class="rename" compatibility="7.6.000" expanded="true" height="82" name="Rename" width="90" x="1452" y="34">
<parameter key="old_name" value="date_1"/>
<parameter key="new_name" value="Date"/>
<list key="rename_additional_attributes"/>
</operator>
<operator activated="true" class="nominal_to_date" compatibility="7.6.000" expanded="true" height="82" name="Nominal to Date" width="90" x="1586" y="34">
<parameter key="attribute_name" value="Date"/>
<parameter key="date_format" value="yyyy-MM-dd"/>
</operator>
<operator activated="true" class="set_role" compatibility="7.6.000" expanded="true" height="82" name="Set Role" width="90" x="1720" y="34">
<parameter key="attribute_name" value="Date"/>
<parameter key="target_role" value="id"/>
<list key="set_additional_roles"/>
</operator>
<connect from_port="in 1" to_op="Data to Documents" to_port="example set"/>
<connect from_op="Data to Documents" from_port="documents" to_op="Combine Documents" to_port="documents 1"/>
<connect from_op="Combine Documents" from_port="document" to_op="JSON To Data" to_port="documents 1"/>
<connect from_op="JSON To Data" from_port="example set" to_op="Numerical to Real" to_port="example set input"/>
<connect from_op="Numerical to Real" from_port="example set output" to_op="De-Pivot" to_port="example set input"/>
<connect from_op="De-Pivot" from_port="example set output" to_op="Split" to_port="example set input"/>
<connect from_op="Split" from_port="example set output" to_op="Replace" to_port="example set input"/>
<connect from_op="Replace" from_port="example set output" to_op="Replace (2)" to_port="example set input"/>
<connect from_op="Replace (2)" from_port="example set output" to_op="Pivot" to_port="example set input"/>
<connect from_op="Pivot" from_port="example set output" to_op="Rename by Replacing" to_port="example set input"/>
<connect from_op="Rename by Replacing" from_port="example set output" to_op="Rename by Replacing (2)" to_port="example set input"/>
<connect from_op="Rename by Replacing (2)" from_port="example set output" to_op="Rename" to_port="example set input"/>
<connect from_op="Rename" from_port="example set output" to_op="Nominal to Date" to_port="example set input"/>
<connect from_op="Nominal to Date" from_port="example set output" to_op="Set Role" to_port="example set input"/>
<connect from_op="Set Role" from_port="example set output" to_port="out 1"/>
<portSpacing port="source_in 1" spacing="0"/>
<portSpacing port="source_in 2" spacing="0"/>
<portSpacing port="sink_out 1" spacing="0"/>
<portSpacing port="sink_out 2" spacing="0"/>
</process>
<description align="center" color="transparent" colored="false" width="126">clean up</description>
</operator>
<connect from_op="Generate Data by User Specification" from_port="output" to_op="Enrich Data by Webservice" to_port="Example Set"/>
<connect from_op="Enrich Data by Webservice" from_port="ExampleSet" to_op="Subprocess" to_port="in 1"/>
<connect from_op="Subprocess" from_port="out 1" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
<description align="center" color="yellow" colored="false" height="201" resized="false" width="640" x="35" y="18">Alpha Vantage API(alphavantage.co)&lt;br&gt;Author: Scott Genzer&lt;br&gt;Published: &amp;#8206;&amp;#8206;8-25-2017&lt;br&gt;Link: http://community.rapidminer.com/t5/RapidMiner-Studio-Knowledge-Base/Real-Time-Financial-Data-via-Alpha-Venture-API-alternative-to/ta-p/41119&lt;br&gt;&lt;br&gt;Note: For each process below, enter your Alpha Vantage API key, ticker symbol and other request elements in the Set Macros operator before running (see https://www.alphavantage.co/documentation/)&lt;br&gt;&lt;br&gt;</description>
</process>
</operator>
</process>

 Feedback welcome.  Enjoy!

 

Scott

«1

Comments

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn

    Wonderful.  Works like a charm!

    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • luc_bartkowskiluc_bartkowski Member Posts: 46 Maven

    Hi Scott,

     

    I implemented your XML into a subprocess and building block. But I don't understand the result sets of this process.

    See the following pictures. Each example set from the Alpha Vantage process contains different attributes. Can you explain why?

     

    TSLA3.jpegFor any subsequent operator the Alpha Vantage process delivers an example set with just 3 attributes

    TSLA2.jpegAlpha Vantage result example set written to CSV delivers a completely other example set.

    TSLA1.jpegAlpha Vantage example set connected to a process output port.

    I don't understand. What is happening here?

    Why does the result example set of the Alpha Vantage process shows 3 types of formatting dependant of the subsequent operator?

     

    Best regards,

    Luc

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    Hello Luc,

     

    I think what is happening here is that the metadata is not being pushed through.  This is typical for Enrich Data via Webservice as RapidMiner has no idea what attributes are coming its way.  It's only picking up the ones that I rename manually afterwards: Date, date_3 and value.  Don't worry - all your attributes are there and you can "select" as many/few as you want.  It's just that you may need to add them manually rather than the nice arrow in Select Attributes.

     

    Screen Shot 2017-10-08 at 9.41.11 PM.png

     

    Scott

  • luc_bartkowskiluc_bartkowski Member Posts: 46 Maven

    Thanks Scott ?

  • luc_bartkowskiluc_bartkowski Member Posts: 46 Maven

    To other users of this building block:

    To get TIME_SERIES_INTRADAY example sets, don't forget to adjust the url REGEX in the "Enrich Data by Webservice (3)" operator that compiles the URL string. For TIME_SERIES_INTRADAY examples sets the required "interval" parameter and related value are neccesary in the url. To accomplish this add an extra Macro "requiredInterval" like this:

     

    requiredInterval.jpegAdd requiredInterval Macro

    And adjust the url REGEX in the "Enrich Data by Webservice (3)" operator like this:

    https://www.alphavantage.co/query?function=%{function}&outputsize=%{outputSize}&symbol=%{tickerSymbol}&apikey=%{apiKey}&interval=%{requiredInterval}

  • websiteguywebsiteguy Member Posts: 24 Maven

    @luc_bartkowski

    Hi Luke , Im a novice, could you tell me please or post a sample how to ajust the date time elements of this process when applying the INTRADAY

    Ive made the changes as specified the process fails. Any help advice appreciated. Regards Lee

    fail.pngfail

  • luc_bartkowskiluc_bartkowski Member Posts: 46 Maven

    Hi Lee,
    Sorry for my late response.
    I'm sorry to say but I didn't continue to use the process of @sgenzer.

     

    I created a database using MySQL for daily and intraday stock prices.

    I wrote a python program that continuously updates this database.

    Accessing stock price data from RapidMiner is then no more than accessing tables/views from this database.

  • websiteguywebsiteguy Member Posts: 24 Maven

    @luc_bartkowski

     

    Hi Luc sorry for my even later reply :)

     

    Do you have an intraday model working I could borrow :/...

     

    I got it working on the daily, but can seem to get it to run on the intraday, prices, mabey its the formatting of the date idk.

     

    regards lee

     

     

  • luc_bartkowskiluc_bartkowski Member Posts: 46 Maven

    Hi Lee,

     

    The thing with the Alpha vantage API is that Intraday returns an extra attribute, the Interval e.g. 5min, 10min, 15min etc. @sgenzer's RM process has to be extended to support this extra parameter.

    I haven't a RM intraday model. I discovered that AV doesn't always return the correct answer, sometimes I receive an API error instead of a resultset. In my python script I included a retry function if such response is received. Secondly: I want to store the data in a database. That requires CRUD functionality that I also included in my python script. Sorry I can't help you in response to your request.

     

  • 1032956310329563 Member Posts: 1 Contributor I

    Hi,

    Is it possible to get step by step tutorial on this?

     

    Kind regards,

    Mareks

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    @10329563 - so it's even easier now than it was when I wrote that article. :) Just do the following:

     

    1. Go to the Alpha Vantage website and get an API key.

    2. Go to the Alpha Vantage Process which is already in the Community Repository by clicking on this link.

    3. Follow these steps:

     

    Screen Shot 2018-10-22 at 10.58.46 AM.png

     

    Screen Shot 2018-10-22 at 10.59.56 AM.png

     

    Click "Apply" and run the process. Done.

     

    Screen Shot 2018-10-22 at 11.03.59 AM.png

     

    Happy RapidMining!

     

    Scott

     

  • cleisnercleisner Member Posts: 4 Contributor I

    Hi, has anyone here ever accessed Accurint (https://secure.accurint.com/app/bps/main) via RapidMiner? This is the best fee-based site for people searches and many other public records. Any tips at all would be greatly appreciated, as I'm just a beginner at this. 

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hi @cleisner - nope not to my knowledge. Happy to collaborate on making it connect if you are a subscriber.

     

    Scott

     

  • cleisnercleisner Member Posts: 4 Contributor I

    Hi, @sgenzer. It looks like I won't be using Accurint after all. It turns out that this is a highly regulated data source and I can't use it as planned.  

  • bjoernbalubjoernbalu Member Posts: 3 Contributor I

    Hello Scott,

    thanks for uploading the process. I still have the problem to get the attributes the useable.

    For example I want to calculate the moving average of the close data. I´m not able to select the data.

    Thanks for your help

    Björn

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hello @bjoernbalu can you post your XML so I can see what you're doing?

  • bjoernbalubjoernbalu Member Posts: 3 Contributor I

    Hello @sgenzer

    here is the XML. The API is removed.

    <?xml version="1.0" encoding="UTF-8"?><process version="9.0.003">
    <context>
    <input/>
    <output/>
    <macros/>
    </context>
    <operator activated="true" class="process" compatibility="9.0.003" expanded="true" name="Process">
    <process expanded="true">
    <operator activated="true" class="set_macros" compatibility="9.0.003" expanded="true" height="68" name="Set Macros" origin="GENERATED_COMMUNITY" width="90" x="112" y="34">
    <list key="macros">
    <parameter key="apiKey" value="API"/>
    <parameter key="tickerSymbol" value="AAPL"/>
    <parameter key="function" value="TIME_SERIES_DAILY_ADJUSTED"/>
    <parameter key="outputSize" value="compact"/>
    <parameter key="datatype" value="csv"/>
    </list>
    <description align="center" color="purple" colored="true" width="126">YOU WILL NEED TO PUT YOUR ALPHA VANTAGE API KEY IN THE FIRST MACRO BEFORE RUNNING!</description>
    </operator>
    <operator activated="true" class="generate_data_user_specification" compatibility="9.0.003" expanded="true" height="68" name="Generate Data by User Specification" origin="GENERATED_COMMUNITY" width="90" x="246" y="34">
    <list key="attribute_values"/>
    <list key="set_additional_roles"/>
    <description align="center" color="transparent" colored="false" width="126">This is blank - ignore me</description>
    </operator>
    <operator activated="true" class="web:enrich_data_by_webservice" compatibility="7.3.000" expanded="true" height="68" name="Enrich Data by Webservice" origin="GENERATED_COMMUNITY" width="90" x="447" y="34">
    <parameter key="query_type" value="Regular Expression"/>
    <list key="string_machting_queries"/>
    <list key="regular_expression_queries">
    <parameter key="foo" value=".*"/>
    </list>
    <list key="regular_region_queries"/>
    <list key="xpath_queries"/>
    <list key="namespaces"/>
    <list key="index_queries"/>
    <list key="jsonpath_queries"/>
    <parameter key="url" value="https://www.alphavantage.co/query?function=%{function}&amp;amp;outputsize=%{outputSize}&amp;amp;symbol=%{tickerSymbol}&amp;amp;apikey=%{apiKey}"/>
    <list key="request_properties"/>
    <description align="center" color="transparent" colored="false" width="126">Alpha Vantage API GET request</description>
    </operator>
    <operator activated="true" class="subprocess" compatibility="9.0.003" expanded="true" height="103" name="Subprocess" origin="GENERATED_COMMUNITY" width="90" x="648" y="34">
    <process expanded="true">
    <operator activated="true" class="text:data_to_documents" compatibility="8.1.000" expanded="true" height="68" name="Data to Documents" origin="GENERATED_COMMUNITY" width="90" x="45" y="34">
    <parameter key="select_attributes_and_weights" value="true"/>
    <list key="specify_weights">
    <parameter key="foo" value="1.0"/>
    </list>
    </operator>
    <operator activated="true" class="text:combine_documents" compatibility="8.1.000" expanded="true" height="82" name="Combine Documents" origin="GENERATED_COMMUNITY" width="90" x="179" y="34"/>
    <operator activated="true" class="text:json_to_data" compatibility="8.1.000" expanded="true" height="82" name="JSON To Data" origin="GENERATED_COMMUNITY" width="90" x="313" y="34"/>
    <operator activated="true" class="numerical_to_real" compatibility="9.0.003" expanded="true" height="82" name="Numerical to Real" origin="GENERATED_COMMUNITY" width="90" x="447" y="34">
    <parameter key="attribute_filter_type" value="regular_expression"/>
    <parameter key="regular_expression" value="Time.*"/>
    </operator>
    <operator activated="true" class="de_pivot" compatibility="9.0.003" expanded="true" height="82" name="De-Pivot" origin="GENERATED_COMMUNITY" width="90" x="581" y="34">
    <list key="attribute_name">
    <parameter key="value" value="Time.*"/>
    </list>
    <parameter key="index_attribute" value="date"/>
    <parameter key="create_nominal_index" value="true"/>
    </operator>
    <operator activated="true" class="split" compatibility="9.0.003" expanded="true" height="82" name="Split" origin="GENERATED_COMMUNITY" width="90" x="715" y="34">
    <parameter key="attribute_filter_type" value="single"/>
    <parameter key="attribute" value="date"/>
    <parameter key="split_pattern" value="[.]\s"/>
    </operator>
    <operator activated="true" class="replace" compatibility="9.0.003" expanded="true" height="82" name="Replace" origin="GENERATED_COMMUNITY" width="90" x="849" y="34">
    <parameter key="attribute_filter_type" value="single"/>
    <parameter key="attribute" value="date_1"/>
    <parameter key="replace_what" value="Time Series \(Daily\)[.]"/>
    </operator>
    <operator activated="true" class="replace" compatibility="9.0.003" expanded="true" height="82" name="Replace (2)" origin="GENERATED_COMMUNITY" width="90" x="983" y="34">
    <parameter key="attribute_filter_type" value="single"/>
    <parameter key="attribute" value="date_1"/>
    <parameter key="replace_what" value="[.][0-9]"/>
    </operator>
    <operator activated="true" class="pivot" compatibility="9.0.003" expanded="true" height="82" name="Pivot" origin="GENERATED_COMMUNITY" width="90" x="1117" y="34">
    <parameter key="group_attribute" value="date_1"/>
    <parameter key="index_attribute" value="date_2"/>
    <parameter key="consider_weights" value="false"/>
    </operator>
    <operator activated="true" class="rename_by_replacing" compatibility="9.0.003" expanded="true" height="82" name="Rename by Replacing" origin="GENERATED_COMMUNITY" width="90" x="1251" y="34">
    <parameter key="replace_what" value="value[_]"/>
    </operator>
    <operator activated="true" class="rename_by_replacing" compatibility="9.0.003" expanded="true" height="82" name="Rename by Replacing (2)" origin="GENERATED_COMMUNITY" width="90" x="1385" y="34">
    <parameter key="replace_what" value="Meta Data[.][0-9][.]\s"/>
    </operator>
    <operator activated="true" class="rename" compatibility="9.0.003" expanded="true" height="82" name="Rename" origin="GENERATED_COMMUNITY" width="90" x="1519" y="34">
    <parameter key="old_name" value="date_1"/>
    <parameter key="new_name" value="Date"/>
    <list key="rename_additional_attributes"/>
    </operator>
    <operator activated="true" class="nominal_to_date" compatibility="9.0.003" expanded="true" height="82" name="Nominal to Date" origin="GENERATED_COMMUNITY" width="90" x="1653" y="34">
    <parameter key="attribute_name" value="Date"/>
    <parameter key="date_format" value="yyyy-MM-dd"/>
    </operator>
    <operator activated="true" class="set_role" compatibility="9.0.003" expanded="true" height="82" name="Set Role" origin="GENERATED_COMMUNITY" width="90" x="1787" y="34">
    <parameter key="attribute_name" value="Date"/>
    <parameter key="target_role" value="id"/>
    <list key="set_additional_roles"/>
    </operator>
    <operator activated="true" class="select_attributes" compatibility="9.0.003" expanded="true" height="82" name="Select Attributes" origin="GENERATED_COMMUNITY" width="90" x="1921" y="34">
    <parameter key="attributes" value="|open|close|low|high"/>
    </operator>
    <operator activated="true" class="multiply" compatibility="9.0.003" expanded="true" height="103" name="Multiply (2)" width="90" x="2055" y="34"/>
    <operator activated="true" class="series:moving_average" compatibility="7.4.000" expanded="true" height="82" name="Moving Average" width="90" x="2189" y="85">
    <parameter key="attribute_name" value="close"/>
    <parameter key="window_width" value="100"/>
    </operator>
    <connect from_port="in 1" to_op="Data to Documents" to_port="example set"/>
    <connect from_op="Data to Documents" from_port="documents" to_op="Combine Documents" to_port="documents 1"/>
    <connect from_op="Combine Documents" from_port="document" to_op="JSON To Data" to_port="documents 1"/>
    <connect from_op="JSON To Data" from_port="example set" to_op="Numerical to Real" to_port="example set input"/>
    <connect from_op="Numerical to Real" from_port="example set output" to_op="De-Pivot" to_port="example set input"/>
    <connect from_op="De-Pivot" from_port="example set output" to_op="Split" to_port="example set input"/>
    <connect from_op="Split" from_port="example set output" to_op="Replace" to_port="example set input"/>
    <connect from_op="Replace" from_port="example set output" to_op="Replace (2)" to_port="example set input"/>
    <connect from_op="Replace (2)" from_port="example set output" to_op="Pivot" to_port="example set input"/>
    <connect from_op="Pivot" from_port="example set output" to_op="Rename by Replacing" to_port="example set input"/>
    <connect from_op="Rename by Replacing" from_port="example set output" to_op="Rename by Replacing (2)" to_port="example set input"/>
    <connect from_op="Rename by Replacing (2)" from_port="example set output" to_op="Rename" to_port="example set input"/>
    <connect from_op="Rename" from_port="example set output" to_op="Nominal to Date" to_port="example set input"/>
    <connect from_op="Nominal to Date" from_port="example set output" to_op="Set Role" to_port="example set input"/>
    <connect from_op="Set Role" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
    <connect from_op="Select Attributes" from_port="example set output" to_op="Multiply (2)" to_port="input"/>
    <connect from_op="Multiply (2)" from_port="output 1" to_port="out 1"/>
    <connect from_op="Multiply (2)" from_port="output 2" to_op="Moving Average" to_port="example set input"/>
    <connect from_op="Moving Average" from_port="example set output" to_port="out 2"/>
    <portSpacing port="source_in 1" spacing="0"/>
    <portSpacing port="source_in 2" spacing="0"/>
    <portSpacing port="sink_out 1" spacing="0"/>
    <portSpacing port="sink_out 2" spacing="0"/>
    <portSpacing port="sink_out 3" spacing="0"/>
    </process>
    <description align="center" color="transparent" colored="false" width="126">JSON parsing and more post-processing</description>
    </operator>
    <connect from_op="Generate Data by User Specification" from_port="output" to_op="Enrich Data by Webservice" to_port="Example Set"/>
    <connect from_op="Enrich Data by Webservice" from_port="ExampleSet" to_op="Subprocess" to_port="in 1"/>
    <connect from_op="Subprocess" from_port="out 1" to_port="result 1"/>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_result 1" spacing="0"/>
    <portSpacing port="sink_result 2" spacing="0"/>
    </process>
    </operator>
    </process>
  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hmm not sure...can you show what the ExampleSet looks like right at that Multiply operator (put a breakpoint).

  • bjoernbalubjoernbalu Member Posts: 3 Contributor I

    Hi,

    there are still 13 regular attributes, but in the list are only 3 attributes (value, date_3, Date) available to use.

    The interesting thing is that everything is working correctly when I storing the data in the repository. After retrieving the attributes are available in the moving average operator. I don´t understand that.

    Thanks for your help and your tutorials.

     

    Björn

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    yes so it is quite common for a RapidMiner process to lose its metadata along the way for various reasons. You can always type the name of the attribute directly in the parameters box even if it does not show up in the pull-down menu:

     

    Screen Shot 2018-10-29 at 4.40.46 PM.png

  • PhilipDPhilipD Member Posts: 2 Contributor I

    Hi, thanks seems to be highly useful..

     

    1) the Alpha Vantage API makes use of other Web Extensions; Web mining and Text processing, be prepared to install those. The Alpha Vantage API appeared in red, it was starightforward enough to click the warning message and follow the steps.

     

    2) When I run it now, only the Date column appears. No price or instrument info. Can't see anything obvious, will try with breakpoints to inspect the data..

     

    Thanks again

    Philip

  • PhilipDPhilipD Member Posts: 2 Contributor I

    Hi, so it looks like the Attributes are not available for selection. Might be a result of the pivot? Don't know enough about this yet to be able to tell. The data is definitely there, right up until Set Role (connect Set Role exa to Out and the result shows exactly what I expect)

     

    I guess I am wondering whether I need the Select Attributes at all.

     

    Will have to see when I try to analyse the data..

     

    Thank you.

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    Thanks @PhilipD. Keep me posted and let me know if I need to make changes to the template.


    Scott

     

  • abevenseeabevensee Member Posts: 4 Contributor I
    @sgenzer seeing the comments about losing metadeta, but I'm having an issue with the Get Pages operator failing when used through web service on RMServer. Process works perfectly in Studio but I want an end user to be able to enter the ticker into a web service and have it run the get the pricing data. Have you seen this before? Any thoughts would be much appreciated.
     
  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    edited February 2020
    hi @abevensee - this is most likely caused by not having the Web Mining extension installed on your job agent.

    Scott
  • SkyTraderSkyTrader Member Posts: 88 Contributor II
    edited June 2020

    Hi there, I’m new to this and machine learning but not new to algorithmic trading. I'm sorry for the many questions here but I had no idea how complicated it would be in getting financial data ready for use with ML.

    Using Excel files and RM:

    I'm assuming that if I want to do an update Excel and get more data from Yahoo Finance (via a Run Web Query) that I will then have to import a newer version of an Excel file into RM and lose the RM work I have on done on eg, an older Excel file that only has data up to May 2020 for example, as opposed to having the very latest data up to the present (10/06/2020)?

    It would seem that it's easier if real time data is being pulled directly into RM rather that using Excel? So I read Scotts post above which contained: 

    "2. Go to the Alpha Vantage Process which is already in the Community Repository by clicking on this link" (I've had to quote it as there is no url to refernece for individual posts?)

    But the link does nothing and the first image I am not sure how to achieve/get to that in RM?

    Neither is there a Alpha Vantage Process in the Marketplace when I search manually?

    Also how do you bring in/call some of those financial indicators on the Alpha Vantage website so that they are alongside the open/High/Low/Close/Volume data? With my Excel solution I will have the O/H/L/C/V data plus the indicator values for each day.

    Also do you know why this website freezes/very slow in Chrome (compared to Firefox where I had to finish off this post)?

    Thanks very much.





  • SkyTraderSkyTrader Member Posts: 88 Contributor II
    edited June 2020
    Hello,
    I was just wondering how regularly these forums are monitored or if I it would have been better to have written a fresh post?
    Thanks.
  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hello @SkyTrader these forums are monitored all the time but it is highly variable whether or not someone has an answer to a specific question.

    As for this post, I will be honest that I have not used AlphaVantage for a long time (pretty much since I wrote that post back in 2017. I know there are a bunch of RM users here on the community who work with financial data...my friend @hughesfleming68 immediately comes to mind. Perhaps he has some thoughts?

    Scott

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    Since you haven't provided either a process or sample data, and have asked a number of questions (some of which are complex), I will provide some general comments.
    RapidMiner processes are designed to be able to be run and rerun very easily. Once you have created a process that takes data in from an underlying source and does additional ETL, it is very simple to rerun that process when you have more data that needs processing. As long as nothing about the underlying data structure has changed, the simple addition of more rows will not inhibit RapidMiner from doing its work efficiently. So losing work on an older workbook is not typically a concern since that work can be easily recreated
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
    I have never used Alpha Vantage but I do use Quandl. Most of these services have well developed python tools. Using those inside an execute python operator is the fastest way to get the data in.
Sign In or Register to comment.