Downloading historical financial data since change of Yahoo API

qwertz2qwertz2 Member Posts: 49 Guru
edited November 2018 in Help

Dear community,

Since Yahoo has changed its API, it is not that easy to access historical financial data anymore by pre-installed operators. Thanks to Thomas who posted a workaround based on the google finance service it is possible to retrieve stock data again.

But unfortunately google finance does not provide CSV download for other object types than stocks. Indices like the Dow Jones, commodities, currencies and bonds are not available.


Is there anyone who knows about free sources of historical data for
- indices
- commodities
- currencies
- bonds
which can be loaded into Rapidminer?



Kind regards
Sachs

Answers

  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,761 Unicorn

    While not an elegant solution (i.e. plug right into RapidMiner), have you looked at YQL? https://developer.yahoo.com/yql/console/

     

    They allow you get an SQL dump of data from different parts of their company.

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn

    Quandl has a lot of financial data---some is free, some is paid, so you'll need to search specifically for what you are looking for, but they have many download formats available, and it can be used reliably with RapidMiner (I have done it).

    https://www.quandl.com/search?query=

     

    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    and I use the API from openexchangerates.com.  Take your pick!  :)

    Scott

  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,761 Unicorn

    @sgenzer that's only for Currencies, right?

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    yes correct - only currencies.  Cost is a freemium model where you get 500(?) calls/month free and pay for more.

  • qwertz2qwertz2 Member Posts: 49 Guru

     

    Hi all,

     

    I found that Yahoo still works when a parameter CRUMB is added. This parameter is individual to your session and stored in cookies. So you could still use Yahoo with but with the drawback that you have to start your browser first and call Yahoo page.

     

    Not yet tested with Rapidminer...

     

     

    Best

    Sachs

  • qwertz2qwertz2 Member Posts: 49 Guru

     

    Ok, this is obviously not as easy as expected...

     

    1) I opened the Yahoo page

    https://finance.yahoo.com/quote/SAN/history?period1=1483250400&period2=1495256400&interval=1d&filter=history&frequency=1d

     

    2) There is a little download data button at the right. This is where I copied that link (which includes the session crumb).

    https://query1.finance.yahoo.com/v7/finance/download/SAN?period1=1483250400&period2=1495256400&interval=1d&events=history&crumb=TAInnNixgdJ

     

    3) This link can be opened from the browser several times. But once opened with Rapidminer it got en error :(

    <?xml version="1.0" encoding="UTF-8"?><process version="7.5.001">
    <context>
    <input/>
    <output/>
    <macros/>
    </context>
    <operator activated="true" class="process" compatibility="7.5.001" expanded="true" name="Process">
    <process expanded="true">
    <operator activated="true" class="open_file" compatibility="7.5.001" expanded="true" height="68" name="Get Online Data as CSV" width="90" x="45" y="34">
    <parameter key="resource_type" value="URL"/>
    <parameter key="filename" value="http://www.google.com/finance/historical?q=NASDAQ:%{symbol}&amp;amp;ei=LRD-WJmhHIOrmgH_4KToAg&amp;amp;output=csv"/>
    <parameter key="url" value="https://query1.finance.yahoo.com/v7/finance/download/SAN?period1=1483250400&amp;period2=1495256400&amp;interval=1d&amp;events=history&amp;crumb=TAInnNixgdJ"/>
    </operator>
    <operator activated="true" class="read_csv" compatibility="7.5.001" expanded="true" height="68" name="Read CSV" width="90" x="179" y="34">
    <parameter key="csv_file" value="C:\Users\Abc\Desktop\bas.csv"/>
    <parameter key="column_separators" value=","/>
    <parameter key="use_quotes" value="false"/>
    <parameter key="date_format" value="dd-MMM-yy"/>
    <parameter key="first_row_as_names" value="false"/>
    <list key="annotations">
    <parameter key="0" value="Name"/>
    </list>
    <parameter key="encoding" value="UTF-8"/>
    <list key="data_set_meta_data_information">
    <parameter key="0" value="Date.true.date.id"/>
    <parameter key="1" value="Open.false.real.attribute"/>
    <parameter key="2" value="High.false.real.attribute"/>
    <parameter key="3" value="Low.false.real.attribute"/>
    <parameter key="4" value="%{symbol_name}.true.real.attribute"/>
    <parameter key="5" value="Volume.false.real.attribute"/>
    </list>
    </operator>
    <connect from_op="Get Online Data as CSV" from_port="file" to_op="Read CSV" to_port="file"/>
    <connect from_op="Read CSV" from_port="output" to_port="result 1"/>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_result 1" spacing="0"/>
    <portSpacing port="sink_result 2" spacing="0"/>
    </process>
    </operator>
    </process>

    This is going to be a looooong night...

     

     

    Cheers

    Sachs

  • qwertz2qwertz2 Member Posts: 49 Guru

     

    So far I could not get the historical data download running in Rapidminer. But as an intermediate result I would like to share three interesting links where other people managed to get it running in different programming languages like Java, Python or VBA:

     

    https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working

    https://stackoverflow.com/questions/44044263/yahoo-finance-historical-data-downloader-url-is-not-working

    https://www.mrexcel.com/forum/excel-questions/1005785-yahoo-finance-api-stock-quotes-changed-3.html

     

    One step closer to the goal for sure but no idea which distance is left on the way to go...

     

    Cheers

    Sachs

  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,761 Unicorn

    Well if they can get it to run in Java, you could build a new extension for it. 

  • qwertz2qwertz2 Member Posts: 49 Guru

     

    Hmm, I tried to get it working in an execute script operator as something to start with. However, sadly I was not able to get it running and I guess I am not the coder with the appropriate knowledge to find the root cause. This is where I have to give in. But hopefully this inspires other members with Java background to continue where I had to stop. Sometimes it's just a colon in the code that messes up everything...

     

    Cheers

    Sachs

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn

    Did you ever check to see whether quandl had the data series you were interested in?  As I mentioned I am able to use their data with no additional coding required.

     

    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • qwertz2qwertz2 Member Posts: 49 Guru

     

    Yes, I tried to but unfortunately the indices data I am looking for is only available in the premium package.

    Thanks for your help anyway :)

     

    Kind regards

    Sachs

  • martinvenichmartinvenich Member Posts: 1 Contributor I

    The spreadsheet here works with Yahoo Finance to download bulk historical data. The VBA can be viewed and modified. The code is more complicated than the now defunct Yahoo API calls, but it still works

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    just to add to this thread...I wrote a KB article two weeks ago about using Alpha Venture API as a good alternative to Yahoo Finance.  You can see the article here: http://community.rapidminer.com/t5/RapidMiner-Studio-Knowledge-Base/Real-Time-Financial-Data-via-Alpha-Venture-API-alternative-to/ta-p/41119

     

    Scott

     

Sign In or Register to comment.