upload directory or Multiple Files to S3

abhi97660abhi97660 Member Posts: 11 Contributor II
edited November 2018 in Help

Hi,

I am using (RM Server and RM Studio) 8.2

I am developing an extension in which I am downloading multiple files (eg: 10 files) sequentially for processing and want to upload processed files to S3 

I want to upload whole directory containing these files to S3.

Do we have any operator to upload Directory to S3...? I tried Write Amazon S3 it takes individual file at once but we can  use loop files operator to do that.

If I want to save upload time, I would like to upload the file as soon as it is downloaded and processed rather than upload all at once.

I tried ouputPort.deliver((IOObject) FileObject) method inside the loop but it is waiting for whole process to get completed.

Any suggestions on how to achieve this simultaneously  in rapidminer where operator is downloading rest of the files and the already downloaded files can be uploaded...?

 

Tagged:

Best Answer

  • Thomas_OttThomas_Ott Posts: 1,761   Unicorn
    Solution Accepted

    @abhi97660 I've never done this before but one idea is to connect the RM SFTP operaters to an S3 bucket. You should be able to load up the entire directory then. 

Sign In or Register to comment.