how to import multiple files

niccayniccay Member Posts: 4 Contributor I
edited November 2018 in Help
Hi,

is there any chance to import multiple files at once? I've got about 70 .csv files having the same scheme that I want to import into a rapidminer repository. Actually, I can't figure out how to solve this problem without any user interaction :-/

My quick and dirty workaround is a little ruby script that
first) reads all filenames of a given directory and
second) creates a rapidminer project-file containing lots of readcsv and store operatores.

I guess, that's not the way you're meant to import multiple files  :)

Answers

  • StaryVenaStaryVena Member Posts: 126  Maven
    Hi,
    you can use "loop files" operator. Than you will probably need "append" operator to merge all example set from collection to one.

    Best,
    Vaclav
    sgenzer
  • mgriffithmgriffith Member Posts: 1 Contributor I
    Can you post a simple example which reads 3 csv files from a single directory and then appends them into one file and stores it.
  • earmijoearmijo Member Posts: 263   Unicorn
    This process reads three csv files from the directory /Users/Ernesto/Desktop/files and appends them.
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="5.2.008">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="5.2.008" expanded="true" name="Process">
        <process expanded="true" height="400" width="659">
          <operator activated="true" class="loop_files" compatibility="5.2.008" expanded="true" height="76" name="Loop Files" width="90" x="246" y="30">
            <parameter key="directory" value="/Users/Ermesto/Desktop/files"/>
            <process expanded="true" height="418" width="677">
              <operator activated="true" class="read_csv" compatibility="5.2.008" expanded="true" height="60" name="Read CSV" width="90" x="112" y="30">
                <parameter key="csv_file" value="/Users/Carlos/Desktop/files/file01.csv"/>
                <parameter key="column_separators" value=","/>
                <parameter key="first_row_as_names" value="false"/>
                <list key="annotations">
                  <parameter key="0" value="Name"/>
                </list>
                <parameter key="encoding" value="MacRoman"/>
                <list key="data_set_meta_data_information">
                  <parameter key="0" value="y.true.integer.attribute"/>
                  <parameter key="1" value="x.true.integer.attribute"/>
                </list>
              </operator>
              <connect from_port="file object" to_op="Read CSV" to_port="file"/>
              <connect from_op="Read CSV" from_port="output" to_port="out 1"/>
              <portSpacing port="source_file object" spacing="0"/>
              <portSpacing port="source_in 1" spacing="0"/>
              <portSpacing port="sink_out 1" spacing="0"/>
              <portSpacing port="sink_out 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="append" compatibility="5.2.008" expanded="true" height="76" name="Append" width="90" x="447" y="30"/>
          <connect from_op="Loop Files" from_port="out 1" to_op="Append" to_port="example set 1"/>
          <connect from_op="Append" from_port="merged set" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
    sgenzer
  • underlinesunderlines Member Posts: 1 Contributor I

    Create the following process:

     

    Loop Files -> Append -> Write CSV

     

    Click on Loop Files, define the parameter directory to point to the directory where your CSV files (or other files readable by rapidminer) are.

    Double click on Loop Files to go into this sub-process.

     

    Create the following sub process:

    fil -> Read CSV -> Select Attributes -> out

     

    Spoiler

    "fil" and "out" are not Operator objects, they are the connectors on the left and right border of the window that look like knobs.

    Click on Select Attributes and select the parameter attribute filter type to either subset or regular_expression.

    For subset, click on the Select Attributes... button, and add the attributes (columns) of your CSVs that you want to have in your merged output. Add them in the right list of the window by typing the name and clicking the plus icon.

    For regular_expression you can define a list of attriutes (columns) like this: .*attribute1.*|.*attribute2.*|.*attribute3.*

    example:

     

    .*mail.*|.*date_submitted.*|.*page_url.*

    Then you are done.

     

     

    Merge columns with different names into the same column (attribute):

    In case you have columns in your CSV with different naming, like: e-mail, eMail, e_mail you can do the following:

    In your existing Select Attributes Object choose regular_expressions. Define a regular_expression that contains all columns you want, and also the variants. If I have the following columns:

    • E_mail
    • e-Mail
    • Mail
    • date_submitted
    • name

    I would create the following regular_expression:

     

     

    .*mail.*|.*date_submitted.*|.*name.*

    This will still create an output with different columns (attribute). To merge the 3 Email columns into one, you have to rename them to be identical. Add a Rename by Replacing Object after the already existing Select Attributes.

     

    On the Rename by Replacing Object select regular_expression as the attribute filter type. Then fill out the fields below like:

    regular expression: .*mail.*

    replace what: .*

    replace by: mail

     

    I attached a gif of my process, to clarify.

    sgenzerEdin_Klapic
  • sgenzersgenzer 12Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,823  Community Manager

    very nice explanation, @underlines !

     

    Scott

     

     

Sign In or Register to comment.