"Saving detail results (folder view) of unsupervised clustering"

B_B_ Member Posts: 70 Maven
edited May 2019 in Help
I've been running unsupervised learning processes and obtaining results.  I would like to save the detail results that show the cluster number and individual items in that cluster (folder view) so I can check the results quickly.  When I use ResultWriter it only writes the Text View (summary) to a file.

Is there another operator or a cluster parameter to set that will save the folder view to a file? 

Or do I have to save the cluster model then run supervised classification to see the detail results?

Thanks.
Tagged:

Answers

  • TobiasMalbrechtTobiasMalbrecht Moderator, Employee, Member Posts: 294 RM Product Management
    Hi,

    as you already noticed, a ResultWriter only saves the textual represenations of the objects that are output of a process. The operator you are looking for is called IOObjectWriter which lets you save any IOObject, in particular also a ClusterModel. Hence, you have to chose ClusterModel for the parameter "io_object" in order to save your ClusterModel. You may the load that ClusterModel back into RapidMiner by using the corresponding IOObjectReader which is in the same operator group (IO.Other).

    Hope that solves your problem.
    Regards,
    Tobias
  • B_B_ Member Posts: 70 Maven
    Tobias

    The IOObjectWriter does save the cluster model as an XML file (and when I load it I see the Cluster Model Data and Folder views) but there is no way to directly use the IO format file. 

    I can write a parser to adapt the XML - this is good enough for now.  Thanks.
  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder
    Hi,

    if you want to show the folder view from your own application (?) maybe reloading it and just using the available cluster model visualization might be an option. This is simply one line for a loaded ClusterModel "cm":

    Component folderView = new ExtendedJScrollPane(new ClusterTreeVisualization((FlatClusterModel) cm));
    Maybe this helps (in case you want to display it in another application).

    Cheers,
    Ingo
  • B_B_ Member Posts: 70 Maven
    Tobias,  Ingo

    While looking for something else I discovered Cluster2ExampleSet.  This does what I need, and then I can write the example set to a database, Excel, etc.

    However, the row/record ID and cluster number are written at the end of each example line, and when I save to Excel the cluster info is dropped.  There is no switch on Exampl2Cluster or ExampleWriter to place the record ID and cluster number at the beginning of the line.  (Over 1800 terms in my small test set, which converts to 1800+ columns to save when the data is written.) 

    Is it possible to add an ExampleWriter parameter to place the important information at the beginning of the line?  Then I can import directly into my database.

    For now I can write a parser to read the end of the line in a csv file and extract what I need.

    Thanks.  (I am sure you guys are being hammered with extra requests from the new release.)

    B.

  • B_B_ Member Posts: 70 Maven
    Perhaps 3 choices of format
    0- as it is now
    1- place rec ID and cluster number at the beginning of the line and attributes after
    2 - output only rec ID and cluster number, leave off cluster attributes
  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder
    Hi,

    well, this is actually already possible. Just use the ExampleSetWriter operator and set the "format" parameter to "special_format" and define an appropriate "special_format" parameter. The documentation (tutorial and the tooltip inside the program or F1) will show you the possible settings for the special format parameter. However, here is the solution for the three suggested scenarios:

    0- as it is now
    Here you can just use the "dense" format:

    <operator name="Root" class="Process" expanded="yes">
        <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
            <parameter key="number_of_attributes" value="2"/>
            <parameter key="target_function" value="gaussian mixture clusters"/>
        </operator>
        <operator name="IdTagging" class="IdTagging">
        </operator>
        <operator name="FeatureNameFilter" class="FeatureNameFilter">
            <parameter key="filter_special_features" value="true"/>
            <parameter key="skip_features_with_name" value="label"/>
        </operator>
        <operator name="KMeans" class="KMeans">
            <parameter key="k" value="4"/>
        </operator>
        <operator name="ClusterModel2ExampleSet" class="ClusterModel2ExampleSet">
            <parameter key="keep_cluster_model" value="false"/>
        </operator>
        <operator name="ExampleSetWriter" class="ExampleSetWriter">
            <parameter key="example_set_file" value="cm_out_complete.dat"/>
        </operator>
    </operator>

    1- place rec ID and cluster number at the beginning of the line and attributes after
    Here you can use the special format "$i $v[cluster] $a":

    <operator name="Root" class="Process" expanded="yes">
        <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
            <parameter key="number_of_attributes" value="2"/>
            <parameter key="target_function" value="gaussian mixture clusters"/>
        </operator>
        <operator name="IdTagging" class="IdTagging">
        </operator>
        <operator name="FeatureNameFilter" class="FeatureNameFilter">
            <parameter key="filter_special_features" value="true"/>
            <parameter key="skip_features_with_name" value="label"/>
        </operator>
        <operator name="KMeans" class="KMeans">
            <parameter key="k" value="4"/>
        </operator>
        <operator name="ClusterModel2ExampleSet" class="ClusterModel2ExampleSet">
            <parameter key="keep_cluster_model" value="false"/>
        </operator>
        <operator name="ExampleSetWriter" class="ExampleSetWriter">
            <parameter key="example_set_file" value="cm_out_complete.dat"/>
            <parameter key="format" value="special_format"/>
            <parameter key="special_format" value="$i $v[cluster] $a"/>
        </operator>
    </operator>

    2 - output only rec ID and cluster number, leave off cluster attributes
    Here you can just omit the parameter "$a" above like in this example:

    <operator name="Root" class="Process" expanded="yes">
        <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
            <parameter key="number_of_attributes" value="2"/>
            <parameter key="target_function" value="gaussian mixture clusters"/>
        </operator>
        <operator name="IdTagging" class="IdTagging">
        </operator>
        <operator name="FeatureNameFilter" class="FeatureNameFilter">
            <parameter key="filter_special_features" value="true"/>
            <parameter key="skip_features_with_name" value="label"/>
        </operator>
        <operator name="KMeans" class="KMeans">
            <parameter key="k" value="4"/>
        </operator>
        <operator name="ClusterModel2ExampleSet" class="ClusterModel2ExampleSet">
            <parameter key="keep_cluster_model" value="false"/>
        </operator>
        <operator name="ExampleSetWriter" class="ExampleSetWriter">
            <parameter key="example_set_file" value="cm_out_complete.dat"/>
            <parameter key="format" value="special_format"/>
            <parameter key="special_format" value="$i $v[cluster]"/>
        </operator>
    </operator>

    Cheers,
    Ingo
  • B_B_ Member Posts: 70 Maven
    OK

    This is what I need.  I'll let you know what the results are.

    Thanks.  Have a good weekend.
  • vijaypshahvijaypshah Member Posts: 30 Maven
    How can I remove the word cluster and only use the number?
    For example instead of having cluster2, i just want to save it as 2

    Regards,
    Vijay
  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder
    Hi,

    I don't get it. I don't get the word "cluster"...

    Could you please post your process (XML from the XML tab) and attach a screenshot showing the place where the word "cluster" occurs? Thanks.

    Cheers,
    Ingo
  • vijaypshahvijaypshah Member Posts: 30 Maven
    I had more time to play around with the tool now. Seems it might have to do with Weka -unsupervised learning maybe, i didn't get the cluster word with kmeans or support vector clustering...

    <operator name="Root" class="Process" expanded="yes">
        <operator name="ExampleSource" class="ExampleSource">
            <parameter key="attributes" value="C:\Documents and Settings\shahv\My Documents\rm_workspace\temp.aml"/>
        </operator>
        <operator name="W-EM" class="W-EM">
            <parameter key="N" value="5.0"/>
        </operator>
    </operator>

    Results aml file

    <?xml version="1.0" encoding="windows-1252"?>
    <attributeset default_source="..\..\Program Files (x86)\Rapid-I\RapidMiner-4.1\temp.dat">

      <attribute
        name        = "feature_id.dat (2)"
        sourcecol    = "1"
        valuetype    = "integer"/>

      <attribute
        name        = "feature_id.dat (3)"
        sourcecol    = "2"
        valuetype    = "integer"/>

      <attribute
        name        = "feature_id.dat (4)"
        sourcecol    = "3"
        valuetype    = "integer"/>

      <attribute
        name        = "feature_id.dat (5)"
        sourcecol    = "4"
        valuetype    = "integer"/>

      <id
        name        = "feature_id.dat (1)"
        sourcecol    = "5"
        valuetype    = "integer"/>

      <cluster
        name        = "cluster"
        sourcecol    = "6"
        valuetype    = "nominal">
          <value>cluster4</value>
          <value>cluster0</value>
          <value>cluster3</value>
          <value>cluster2</value>
          <value>cluster1</value>
      </cluster>

    </attributeset>

    [attachment deleted by admin]
  • B_B_ Member Posts: 70 Maven
    Ingo

    The Special Format output is working.  I'm able to get just the record ID and cluster number. 

    Thanks for your help.

    B.
Sign In or Register to comment.