"Reuse of already opened sql connection"

RLynxRLynx Member Posts: 18 Contributor II
edited June 2019 in Help
I Loop over big number of values of specific attribute. On each iteration I get additional data from particular (always the same database) and produce calculation results. If it goes too fast Rapid Miner eventually can't open new connection to that database, all ports are in use... so, I have to introduce some kind of delay in order to grab new data from database.. I think, it would be more practical to save and re-use already opened connection... Question: how we can do it?

I do have additional workaround for this problem: Use Remember operator and do filtering on each iteration step... but this involves more memory and time consumption - size of overall data is huge.

Thank you for you comments and suggestions!
Tagged:

Answers

  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Hi,

    the problem why RapidMiner can't reconnect to the server is, as you stated above, that the connection limit on the server has been reached. That happens because of a bug which did not close the connections the server after use. This issue is fixed in current svn and will be released with the next update. Eventually we will also implement reusing connections, but currently we don't support that.

    As a workaround you might find a possiblity to split up your many small queries into a larger one and do additional filtering in RapidMiner, as you are currently doing. How big the "big" query should be depends on the process - you have a trade-off between rather slow client side filtering and the overhead of querying the sql server many times.

    Regards,
    Marius
  • RLynxRLynx Member Posts: 18 Contributor II
    Thank you for good news about an update and good workaround suggestion.

    best regards,
    RLynx
Sign In or Register to comment.