how to import multiple files

niccay

Hi,

is there any chance to import multiple files at once? I've got about 70 .csv files having the same scheme that I want to import into a rapidminer repository. Actually, I can't figure out how to solve this problem without any user interaction :-/

My quick and dirty workaround is a little ruby script that
first) reads all filenames of a given directory and
second) creates a rapidminer project-file containing lots of readcsv and store operatores.

I guess, that's not the way you're meant to import multiple files  

StaryVena

Hi,
you can use "loop files" operator. Than you will probably need "append" operator to merge all example set from collection to one.

Best,
Vaclav

mgriffith

Can you post a simple example which reads 3 csv files from a single directory and then appends them into one file and stores it.

earmijo

This process reads three csv files from the directory /Users/Ernesto/Desktop/files and appends them.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.2.008">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.2.008" expanded="true" name="Process">
    <process expanded="true" height="400" width="659">
      <operator activated="true" class="loop_files" compatibility="5.2.008" expanded="true" height="76" name="Loop Files" width="90" x="246" y="30">
        <parameter key="directory" value="/Users/Ermesto/Desktop/files"/>
        <process expanded="true" height="418" width="677">
          <operator activated="true" class="read_csv" compatibility="5.2.008" expanded="true" height="60" name="Read CSV" width="90" x="112" y="30">
            <parameter key="csv_file" value="/Users/Carlos/Desktop/files/file01.csv"/>
            <parameter key="column_separators" value=","/>
            <parameter key="first_row_as_names" value="false"/>
            <list key="annotations">
              <parameter key="0" value="Name"/>
            </list>
            <parameter key="encoding" value="MacRoman"/>
            <list key="data_set_meta_data_information">
              <parameter key="0" value="y.true.integer.attribute"/>
              <parameter key="1" value="x.true.integer.attribute"/>
            </list>
          </operator>
          <connect from_port="file object" to_op="Read CSV" to_port="file"/>
          <connect from_op="Read CSV" from_port="output" to_port="out 1"/>
          <portSpacing port="source_file object" spacing="0"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="append" compatibility="5.2.008" expanded="true" height="76" name="Append" width="90" x="447" y="30"/>
      <connect from_op="Loop Files" from_port="out 1" to_op="Append" to_port="example set 1"/>
      <connect from_op="Append" from_port="merged set" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

underlines

Create the following process:

 

Loop Files -> Append -> Write CSV

 

Click on Loop Files, define the parameter directory to point to the directory where your CSV files (or other files readable by rapidminer) are.

Double click on Loop Files to go into this sub-process.

 

Create the following sub process:

fil -> Read CSV -> Select Attributes -> out

 

Spoiler

"fil" and "out" are not Operator objects, they are the connectors on the left and right border of the window that look like knobs.

Click on Select Attributes and select the parameter attribute filter type to either subset or regular_expression.

For subset, click on the Select Attributes... button, and add the attributes (columns) of your CSVs that you want to have in your merged output. Add them in the right list of the window by typing the name and clicking the plus icon.

For regular_expression you can define a list of attriutes (columns) like this: .*attribute1.*|.*attribute2.*|.*attribute3.*

example:

 

.*mail.*|.*date_submitted.*|.*page_url.*

Then you are done.

 

 

Merge columns with different names into the same column (attribute):

In case you have columns in your CSV with different naming, like: e-mail, eMail, e_mail you can do the following:

In your existing Select Attributes Object choose regular_expressions. Define a regular_expression that contains all columns you want, and also the variants. If I have the following columns:

• E_mail
• e-Mail
• Mail
• date_submitted
• name

I would create the following regular_expression:

 

 

.*mail.*|.*date_submitted.*|.*name.*

This will still create an output with different columns (attribute). To merge the 3 Email columns into one, you have to rename them to be identical. Add a Rename by Replacing Object after the already existing Select Attributes.

 

On the Rename by Replacing Object select regular_expression as the attribute filter type. Then fill out the fields below like:

regular expression: .*mail.*

replace what: .*

replace by: mail

 

I attached a gif of my process, to clarify.

sgenzer

very nice explanation, @underlines !

 

Scott