Retrieve file over SFTP conection

frankie

Hello,

how can this be done? I realize that you can execute scripts, but are there any built in tricks in RM that can perform this kind of operation?
Does anybody have code to share where this problem has been tackled one way or another?  ::)

Also, can I somehow provide the name of the file to be retrieved dynamically to the script/operator?


Thank you,
frankie

IngoRM

Hi Frankie,

hmm, I must admit that I am not really sure if this works without any external tool, sorry. Might be but than I am not aware of that.

About providing the name dynamically: You could use macros for that. Just define the macro in your process or extract it from your data. You can then use it everywhere via %{your_macro_name}. There are lots of example for this on myExperiment which can be downloaded with our Community Extension for RapidMiner.

Cheers,
Ingo

Jugi

Hi,

 

Any News about SFTP access in the last 5 years?

 

Regards

Julian

robin

bump.

sgenzer

hello @robin - feel free to post in Ideas -> Product Ideas. People can upvote, etc.. there.

 

Scott

 

 

kayman

Not the most straightforward way but I do this using python.

There are plenty of sftp libraries available, most have decent copy paste examples so you don't need to be a coding geek to get it running.

 

Below is a basic script using ftplib to upload a file. Unfortunatly it isn't sftp (can't find back immediatly where I used it) but the principle remains the same.

 

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="UTF-8"/>
    <process expanded="true">
      <operator activated="true" class="set_macros" compatibility="7.6.001" expanded="true" height="68" name="file variables" width="90" x="112" y="34">
        <list key="macros">
          <parameter key="from" value="%{your_source_path}"/>
          <parameter key="to" value="%{your_destination_path}"/>
          <parameter key="filename" value="%{your_file}"/>
        </list>
      </operator>
      <operator activated="true" class="python_scripting:execute_python" compatibility="7.4.000" expanded="true" height="68" name="upload file" width="90" x="246" y="34">
        <parameter key="script" value="import pandas as pd&#10;from ftplib import FTP&#10;&#10;def rm_main():&#10;&#10;    ftp = FTP('ftp_address')&#10;    #ftp.login(user='username', passwd = 'password')&#10;    ftp.login()&#10;    target = &quot;%{from}/%{filename}&quot;&#10;    ftp.cwd(&quot;%{to}&quot;)&#10;    &#10;    try:&#10;        f = open(target, &quot;rb&quot;)&#10;        ftp.storbinary('STOR '+'%{filename}', f)&#10;        f.close()&#10;        ftp.quit()        &#10;        print(&quot;success&quot;)&#10;&#10;    except:&#10;        print(&quot;failed&quot;)&#10;    return "/>
      </operator>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
    </process>
  </operator>
</process>

and for download

 

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="UTF-8"/>
    <process expanded="true">
      <operator activated="true" class="set_macros" compatibility="7.6.001" expanded="true" height="68" name="file variables" width="90" x="112" y="34">
        <list key="macros">
          <parameter key="from" value="your_sourcepath"/>
          <parameter key="to" value="your_destination_path"/>
          <parameter key="filename" value="sample.txt"/>
        </list>
      </operator>
      <operator activated="true" class="python_scripting:execute_python" compatibility="7.4.000" expanded="true" height="68" name="download file" width="90" x="246" y="34">
        <parameter key="script" value="import pandas as pd&#10;import os&#10;&#10;from ftplib import FTP&#10;&#10;def rm_main():&#10;&#10;    ftp = FTP('your_ftp_address')&#10;    #ftp.login(user='username', passwd = 'password')&#10;    ftp.login()&#10;&#9;&#10;    ftp.cwd(&quot;%{from}&quot;)&#10;    filename = &quot;%{filename}&quot;&#10;    &#10;    target = os.path.join(&quot;%{to}&quot;,filename)&#10;    &#10;    try:&#10;        f = open(target, 'wb')&#10;        ftp.retrbinary('RETR ' + filename, f.write)&#10;        f.close&#10;        ftp.quit() &#10;        &#10;        print(&quot;success&quot;)&#10;        &#10;    except:&#10;    &#10;        print(&quot;failed&quot;)&#10;        &#10;    return "/>
      </operator>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
    </process>
  </operator>
</process>

Look for sftp examples and change the code accordingly, ask your local python guru for support if needed but it's less scary as it looks at first glance