RapidMiner 9.7 is Now Available

Lots of amazing new improvements including true version control! Learn more about what's new here.

CLICK HERE TO DOWNLOAD

Loop until the value of two macros are equal?

ZKuiperZKuiper Member Posts: 11 Contributor II
edited March 25 in Help
Hello, I am looking to pull a large amount of data from a server which limits how many points i can pull each time i access it. I would like to get around this by looping the pull and building the table iteratively. 

Below is some pseudo code of what i'd like to do:

INSIDE LOOP
%{Moving End} = IF %{Start Time} + %{Step Size} < %{Final End}
                                  THEN %{Start Time} + %{Step Size}
                                   ELSE %{Final End}
Execute Data Pull Block
Append New Pull to Table
%{Start Time} = %{Moving End}

BREAK IF %{Start Time}=%{Final End}

Any help on how to tell the loop to break on this condition would be appreciated, thanks!.

Best Answer

Answers

  • yyhuangyyhuang Administrator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 273  RM Data Scientist
    Hi Zak,

    Are you trying to pull data  in batches from OSI server (cc. @Michael )? You can list the timestamps for start time and end time in a reference table and apply "loop values" with macro.



    Let me know if you have followup questions..

    Cheers,
    YY
  • ZKuiperZKuiper Member Posts: 11 Contributor II
    Yeah, I am trying to pull from the OSI server. I'll give this a shot tomorrow morning before we speak and see how it works, thanks!
  • MichaelMichael Administrator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 18  RM Data Scientist
    Hi Zak,
    We have been discussing adding support for client-side batching internally, but have so far not given it high priority. Reason being that we are simply not sure how often users would run into the limits given that the PI System allows to do common pre-processing steps on the server side (e.g., creating compact equidistant time series).
    Do you consider pulling more data than the server allows in a single request a common scenario?
    Best,
    Michael
    yyhuang
  • ZKuiperZKuiper Member Posts: 11 Contributor II
    So a medium standard data dig for one of our engineers i'd estimate to be like 25 tags, 1 hour resolution over a year = 219,000 data points which exceeds the 150,000 max. I am working on bigger pulls so mine are in the realm of 50 - 100 million data points. From my view it would be common.

    As a side note I have figured out how to automatically add in a tag list from excel using the "Set Macros from Example Set" and it works well so that ask can fall down the list a bit.
    mschmitzMichaelyyhuang
  • MichaelMichael Administrator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 18  RM Data Scientist
    Thanks for the insights. Guess we have to revisit the client-side batching. 
    The side note is interesting, too. A simple way to provide similar functionality without having to use macros might be to add an optional input that if connected replaces the data item parameters (just as the connection input replaces the connection parameter).
    yyhuang
Sign In or Register to comment.