Options

Retrieve file over SFTP conection

frankiefrankie Member Posts: 26 Contributor II
edited November 2018 in Help
Hello,

how can this be done? I realize that you can execute scripts, but are there any built in tricks in RM that can perform this kind of operation?
Does anybody have code to share where this problem has been tackled one way or another?  ::)

Also, can I somehow provide the name of the file to be retrieved dynamically to the script/operator?


Thank you,
frankie




Answers

  • Options
    IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder
    Hi Frankie,

    hmm, I must admit that I am not really sure if this works without any external tool, sorry. Might be but than I am not aware of that.

    About providing the name dynamically: You could use macros for that. Just define the macro in your process or extract it from your data. You can then use it everywhere via %{your_macro_name}. There are lots of example for this on myExperiment which can be downloaded with our Community Extension for RapidMiner.

    Cheers,
    Ingo
  • Options
    JugiJugi RapidMiner Certified Analyst, Member Posts: 12 Contributor II

    Hi,

     

    Any News about SFTP access in the last 5 years?

     

    Regards

    Julian

  • Options
    robinrobin Member Posts: 100 Guru

    bump.

  • Options
    sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hello @robin - feel free to post in Ideas -> Product Ideas. People can upvote, etc.. there.

     

    Scott

     

     

  • Options
    kaymankayman Member Posts: 662 Unicorn

    Not the most straightforward way but I do this using python.

    There are plenty of sftp libraries available, most have decent copy paste examples so you don't need to be a coding geek to get it running.

     

    Below is a basic script using ftplib to upload a file. Unfortunatly it isn't sftp (can't find back immediatly where I used it) but the principle remains the same.

     

     

    <?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
        <parameter key="logverbosity" value="init"/>
        <parameter key="random_seed" value="2001"/>
        <parameter key="send_mail" value="never"/>
        <parameter key="notification_email" value=""/>
        <parameter key="process_duration_for_mail" value="30"/>
        <parameter key="encoding" value="UTF-8"/>
        <process expanded="true">
          <operator activated="true" class="set_macros" compatibility="7.6.001" expanded="true" height="68" name="file variables" width="90" x="112" y="34">
            <list key="macros">
              <parameter key="from" value="%{your_source_path}"/>
              <parameter key="to" value="%{your_destination_path}"/>
              <parameter key="filename" value="%{your_file}"/>
            </list>
          </operator>
          <operator activated="true" class="python_scripting:execute_python" compatibility="7.4.000" expanded="true" height="68" name="upload file" width="90" x="246" y="34">
            <parameter key="script" value="import pandas as pd&#10;from ftplib import FTP&#10;&#10;def rm_main():&#10;&#10;    ftp = FTP('ftp_address')&#10;    #ftp.login(user='username', passwd = 'password')&#10;    ftp.login()&#10;    target = &quot;%{from}/%{filename}&quot;&#10;    ftp.cwd(&quot;%{to}&quot;)&#10;    &#10;    try:&#10;        f = open(target, &quot;rb&quot;)&#10;        ftp.storbinary('STOR '+'%{filename}', f)&#10;        f.close()&#10;        ftp.quit()        &#10;        print(&quot;success&quot;)&#10;&#10;    except:&#10;        print(&quot;failed&quot;)&#10;    return "/>
          </operator>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
        </process>
      </operator>
    </process>

    and for download

     

     

    <?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
        <parameter key="logverbosity" value="init"/>
        <parameter key="random_seed" value="2001"/>
        <parameter key="send_mail" value="never"/>
        <parameter key="notification_email" value=""/>
        <parameter key="process_duration_for_mail" value="30"/>
        <parameter key="encoding" value="UTF-8"/>
        <process expanded="true">
          <operator activated="true" class="set_macros" compatibility="7.6.001" expanded="true" height="68" name="file variables" width="90" x="112" y="34">
            <list key="macros">
              <parameter key="from" value="your_sourcepath"/>
              <parameter key="to" value="your_destination_path"/>
              <parameter key="filename" value="sample.txt"/>
            </list>
          </operator>
          <operator activated="true" class="python_scripting:execute_python" compatibility="7.4.000" expanded="true" height="68" name="download file" width="90" x="246" y="34">
            <parameter key="script" value="import pandas as pd&#10;import os&#10;&#10;from ftplib import FTP&#10;&#10;def rm_main():&#10;&#10;    ftp = FTP('your_ftp_address')&#10;    #ftp.login(user='username', passwd = 'password')&#10;    ftp.login()&#10;&#9;&#10;    ftp.cwd(&quot;%{from}&quot;)&#10;    filename = &quot;%{filename}&quot;&#10;    &#10;    target = os.path.join(&quot;%{to}&quot;,filename)&#10;    &#10;    try:&#10;        f = open(target, 'wb')&#10;        ftp.retrbinary('RETR ' + filename, f.write)&#10;        f.close&#10;        ftp.quit() &#10;        &#10;        print(&quot;success&quot;)&#10;        &#10;    except:&#10;    &#10;        print(&quot;failed&quot;)&#10;        &#10;    return "/>
          </operator>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
        </process>
      </operator>
    </process>

    Look for sftp examples and change the code accordingly, ask your local python guru for support if needed but it's less scary as it looks at first glance

     

Sign In or Register to comment.