How to get index of a word in the list and can we extract the value based on index?

AnushaAnusha Member Posts: 19 Maven
Hi All!

My dataset has six columns. All columns are in the list format.
first column(list) has common words which are present in 2nd(ch_1) and 3rd(ch_2) columns. The respective values are present in 4th(va_1) and 5th(va_2) columns. So, I need to loop over the each word present in "list" column and extract the index of that particular in ch_1& ch_2 and get the respective values in va_1&va_2. After getting va_1 and va_2, I have to compare if va_1 not equal to va_2 flag "mis" value becomes "1" else "0".


list                          ch_1                         ch_2                  va_1                        va_2
typ,shap               typ,shap,fgh            typ,fgi,shap           8,"ghk",90               8,97, "oki" 

----here typ,shap are common words in ch_1&ch_2 and their respective indexes are (0,1)in ch_1 and (0,2) in ch_2. so, I need values from va_1&va_2 based on indexes. As (0,1) in ch_1, need to extract (0,1) values from va_1 and in the same way (0,2) are indexes of ch_2, need to get values of (0,2) from va_2.

Required output:

list                          ch_1                         ch_2                  va_1                        va_2                         mis
typ,shap               typ,shap,fgh            typ,fgi,shap           8,"ghk",90               8,97, "oki"                  1

----"mis" value is "1" because for the "typ"----va_1=8 and va_2=8 matched in the case.
But for the "shap"----va_1="ghk" and va_2="oki", here the values are mismatched. so, If one value is mismatched, then the flag should be "1".

Can anyone help me?

Thanks in Advance!


  • Options
    MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,517 RM Data Scientist
    Hi there,

    the first step is to convert your data set into a proper data frame. I recommend to do this in the ETL process which pushes the data into your data lake. This is 100% standard thing to do.

    Attached is a process to convert it into a format we may use more easily. I think afterwards its just another generate Attributes + maybe pivot it back.


    <?xml version="1.0" encoding="UTF-8"?><process version="9.9.000">
      <operator activated="true" class="process" compatibility="9.9.000" expanded="true" name="Process">
        <parameter key="logverbosity" value="init"/>
        <parameter key="random_seed" value="2001"/>
        <parameter key="send_mail" value="never"/>
        <parameter key="notification_email" value=""/>
        <parameter key="process_duration_for_mail" value="30"/>
        <parameter key="encoding" value="SYSTEM"/>
        <process expanded="true">
          <operator activated="true" class="utility:create_exampleset" compatibility="9.9.000" expanded="true" height="68" name="Create ExampleSet" width="90" x="45" y="34">
            <parameter key="generator_type" value="comma separated text"/>
            <parameter key="number_of_examples" value="100"/>
            <parameter key="use_stepsize" value="false"/>
            <list key="function_descriptions"/>
            <parameter key="add_id_attribute" value="false"/>
            <list key="numeric_series_configuration"/>
            <list key="date_series_configuration"/>
            <list key="date_series_configuration (interval)"/>
            <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss"/>
            <parameter key="time_zone" value="SYSTEM"/>
            <parameter key="input_csv_text" value="list;ch_1;ch_2;va_1;va_2&#10;typ,shap;typ,shap,fgh;typ,fgi,shap;  8,&quot;ghk&quot;,90;8,97, &quot;oki&quot;  "/>
            <parameter key="column_separator" value=";"/>
            <parameter key="parse_all_as_nominal" value="true"/>
            <parameter key="decimal_point_character" value="."/>
            <parameter key="trim_attribute_names" value="true"/>
          <operator activated="true" class="generate_id" compatibility="9.9.000" expanded="true" height="82" name="Generate ID" width="90" x="380" y="34">
            <parameter key="create_nominal_ids" value="false"/>
            <parameter key="offset" value="0"/>
          <operator activated="true" class="split" compatibility="9.9.000" expanded="true" height="82" name="Split" width="90" x="514" y="34">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attribute" value=""/>
            <parameter key="attributes" value="va_1|va_2"/>
            <parameter key="use_except_expression" value="false"/>
            <parameter key="value_type" value="nominal"/>
            <parameter key="use_value_type_exception" value="false"/>
            <parameter key="except_value_type" value="file_path"/>
            <parameter key="block_type" value="single_value"/>
            <parameter key="use_block_type_exception" value="false"/>
            <parameter key="except_block_type" value="single_value"/>
            <parameter key="invert_selection" value="false"/>
            <parameter key="include_special_attributes" value="false"/>
            <parameter key="split_pattern" value=","/>
            <parameter key="split_mode" value="ordered_split"/>
          <operator activated="true" class="de_pivot" compatibility="9.9.000" expanded="true" height="82" name="De-Pivot" width="90" x="648" y="34">
            <list key="attribute_name">
              <parameter key="first_list" value="va_1_(\d+)"/>
              <parameter key="secondlist" value="va_2_(\d+)"/>
            <parameter key="index_attribute" value="rowId"/>
            <parameter key="create_nominal_index" value="false"/>
            <parameter key="keep_missings" value="false"/>
          <connect from_op="Create ExampleSet" from_port="output" to_op="Generate ID" to_port="example set input"/>
          <connect from_op="Generate ID" from_port="example set output" to_op="Split" to_port="example set input"/>
          <connect from_op="Split" from_port="example set output" to_op="De-Pivot" to_port="example set input"/>
          <connect from_op="De-Pivot" from_port="example set output" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>

    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Options
    AnushaAnusha Member Posts: 19 Maven
    Hi @mschmitz,

    Sorry. Maybe you didn't get my question right. For your understanding, I'm twisting my data set as key-value pairs.

    Source:("list" attribute has words that are common in other two attributes ch_val_1 & ch_val_2)

    list                        ch_val_1                               ch_val_2
    typ,shap            shap: "cir",coun:1,typ:23          jkl:90,shap:"cir",typ:32
    fit                        coun:4,fit:23                             fit:23,jkl:09

    Output required:

    list                        ch_val_1                               ch_val_2                                     mis
    typ,shap            shap: "cir",coun:1,typ:23          jkl:90,shap:"cir",typ:32                1
    fit                        coun:4,fit:23                             fit:23,jkl:09                                 0

    In first record, "typ" value in ch_val_1 attribute is 23 and in ch_val_2 attribute, it is 32. so, there is a mismatch of values. Hence mis flag should be "1". Even though second-word "shap" value is matched between ch_val_1 & ch_val_2. if one value is mismatched in that row flag should be highlighted.
    In the second record,"fit" word value in ch_val_1 is 23 and ch_val_2 also it's 23. so as there is no mismatch, flag should be "0".

    Could you please help me?

  • Options
    Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist
    this looks quite similar to your problem in this post: https://community.rapidminer.com/discussion/comment/69757
    If you use the "Split" Operator with the separator "," you should receive an ExampleSet which looks like the one mentioned in above post.
    Then you could use the solution created there.
    Happy Mining and Stay safe,

Sign In or Register to comment.