"Determine text similarity?"

Legacy UserLegacy User Member Posts: 0 Newbie
edited May 23 in Help
Hi,

is it possible to use RapidMiner to determine the similarity of two texts (i.e. using cosing similarity)?

I played around with RapidMiner and the text plugin. I managed to create word vectors using TextInput and applied StringTokenizer, EnglishStopwordFilter and PorterStemmer.

But now I'm stuck. How can I compare two text files and determine their similarity?

I'm thankful for any hint!

Answers

  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,597  RM Founder
    Hi,

    did you try out the operator ExampleSet2Similarity? If you search for "similarity" in the field below the operator groups in the "New Operator" tab or in the text field of the "New Operator" dialog, this (and other similarity related) operator should come up...

    Cheers,
    Ingo
  • Legacy UserLegacy User Member Posts: 0 Newbie
    Hi,
    thanks for your answer.

    I tried that, but will give me an illegal argument exception: null or zero length argument @ ExampleSet2Similarity

    Howerver, within DataStatistics, I have output which looks like this:
    #1: authorit (real/single_value): avg = 0.09033030954895138 +/- 0.0; unknown = 0.0 #2: sourc (real/single_value): avg = 0.30110103182983794 +/- 0.0; unknown = 0.0 #3: data (real/single_value): avg = 0.5419818572937083 +/- 0.0; unknown = 0.0 #4: alia (real/single_value): avg = 0.030110103182983794 +/- 0.0; unknown = 0.0 #5: system (real/single_value): avg = 0.030110103182983794 +/- 0.0; unknown = 0.0 #6: record (real/single_value): avg = 0.06022020636596759 +/- 0.0; unknown = 0.0 #7: trust (real/single_value): avg = 0.06022020636596759 +/- 0.0; unknown = 0.0 #8: motiv (real/single_value): avg = 0.030110103182983794 +/- 0.0; unknown = 0.0 #9: patient (real/single_value): avg = 0.030110103182983794 +/- 0.0; unknown = 0.0 #10: heath (real/single_value): avg = 0.030110103182983794 +/- 0.0; unknown = 0.0
    My process looks like this:
    image
  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,597  RM Founder
    Hello,

    hmm, I just tried that myself with a data set delivered together with the Text plugin and everythings seems to work normally. Here is the process:

    <operator name="Root" class="Process" expanded="yes">
        <operator name="TextInput" class="TextInput" expanded="yes">
            <list key="texts">
              <parameter key="graphics" value="../data/newsgroup/graphics"/>
              <parameter key="hardware" value="../data/newsgroup/hardware"/>
            </list>
            <parameter key="default_content_encoding" value="ISO-8859-1"/>
            <parameter key="prune_below" value="2"/>
            <list key="namespaces">
            </list>
            <parameter key="create_text_visualizer" value="true"/>
            <parameter key="on_the_fly_pruning" value="3"/>
            <operator name="StringTokenizer" class="StringTokenizer">
            </operator>
            <operator name="EnglishStopwordFilter" class="EnglishStopwordFilter">
            </operator>
            <operator name="TokenLengthFilter" class="TokenLengthFilter">
                <parameter key="min_chars" value="3"/>
            </operator>
            <operator name="ToLowerCaseConverter" class="ToLowerCaseConverter">
            </operator>
            <operator name="TermNGramGenerator" class="TermNGramGenerator">
            </operator>
        </operator>
        <operator name="ExampleSet2Similarity" class="ExampleSet2Similarity">
        </operator>
    </operator>
    Cheers,
    Ingo
Sign In or Register to comment.