RapidMiner

Search for replies to a particular twitter handle

SOLVED
Regular Contributor

Search for replies to a particular twitter handle

Hello again,

Is there a way to configure any of the available twitter operators to look for replies to a twitter handle?? Thank you very much.

5 REPLIES
RMStaff

Re: Search for replies to a particular twitter handle

Hi Batstache,

you can use the regular "Search Twitter" Operator to search for tweets containing the query @username to find tweets mentioning the given user.

Finding tweets mentioning the user rapidminerFinding tweets mentioning the user rapidminerResult view for the 10 most recent resultsResult view for the 10 most recent results

Here is a possible process. Keep in mind, that you need to configure a twitter connection over the "Connections" tab first. Here I'm using a connection I named "Twitter Standard".

<?xml version="1.0" encoding="UTF-8"?><process version="7.5.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.5.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="983" y="34">
        <parameter key="connection" value="Twitter Standard"/>
        <parameter key="query" value="@rapidminer"/>
        <parameter key="result_type" value="recent"/>
        <parameter key="limit" value="10"/>
      </operator>
      <connect from_op="Search Twitter" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Cheers,

Philipp

Elite III

Re: Search for replies to a particular twitter handle

Thanks for the sample process! Is there any way to directly limit the return to replies, or is it necessary to do some text searching afterwards for that?
Brian T., Lindon Ventures - www.lindonventures.com
Analytics Consulting by Certified RapidMiner Analysts
Regular Contributor

Re: Search for replies to a particular twitter handle

Thank you @pschlunder, I was using this setup already. But I specifically wanted to look for replies to tweets only and not retweets. I CAN filter out the retweets using RegEx, but I was hoping that similar to the operator that gets tweets from a particular handle, there might be one that gets all replies to those tweets. Perhaps I don't know very well how twitter works since I do not use it or Facebook for that matter. But thank you for your help.

RMStaff

Re: Search for replies to a particular twitter handle

[ Edited ]

There is a column called "To-User-Id" (as well as "To-User" for the actual username). This one is set, if the tweet is regarded as a reply. Hence if you want to ignore Retweets make sure, that this given field is not empty. Furthermore it seems, that since it only contains one id, that only the person metioned first in the tweet is regarded as the person replied to. In general a reply in twitter is invoked by starting the tweet with "@username" followed by your message.

 

Since there is no extra operator, yet. I've created a subprocess you can use for the time being.

 

Get Mentions for a list of Twitter UsersGet Mentions for a list of Twitter Users

<?xml version="1.0" encoding="UTF-8"?><process version="7.5.001">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="7.5.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="text:create_document" compatibility="7.5.000" expanded="true" height="68" name="List of Users" width="90" x="112" y="34">
<parameter key="text" value="rapidminer"/>
<description align="center" color="transparent" colored="false" width="126">Provide a list of usernames as a document</description>
</operator>
<operator activated="true" class="set_macro" compatibility="7.5.001" expanded="true" height="82" name="Number of Tweets" width="90" x="112" y="187">
<parameter key="macro" value="tweets_to_search_through"/>
<parameter key="value" value="100"/>
<description align="center" color="transparent" colored="false" width="126">set amount of tweets to scan through for mentions</description>
</operator>
<operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Get Mentions from" width="90" x="313" y="34">
<process expanded="true">
<operator activated="true" class="text:documents_to_data" compatibility="7.5.000" expanded="true" height="82" name="Documents to Data" width="90" x="313" y="34">
<parameter key="text_attribute" value="username"/>
</operator>
<operator activated="true" class="concurrency:loop_values" compatibility="7.5.001" expanded="true" height="82" name="Loop Values" width="90" x="514" y="34">
<parameter key="attribute" value="username"/>
<parameter key="iteration_macro" value="username"/>
<process expanded="true">
<operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details" width="90" x="45" y="238">
<parameter key="connection" value="Twitter Standard"/>
<parameter key="user" value="%{username}"/>
<description align="center" color="transparent" colored="false" width="126">get internal twitter id of the user for more accurate matching</description>
</operator>
<operator activated="true" class="extract_macro" compatibility="7.5.001" expanded="true" height="68" name="Extract Macro" width="90" x="179" y="238">
<parameter key="macro" value="user-id"/>
<parameter key="macro_type" value="data_value"/>
<parameter key="attribute_name" value="Id"/>
<parameter key="example_index" value="1"/>
<list key="additional_macros"/>
<description align="center" color="transparent" colored="false" width="126">make it available for future filtering</description>
</operator>
<operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="45" y="34">
<parameter key="connection" value="Twitter Standard"/>
<parameter key="query" value="@%{username}"/>
<parameter key="result_type" value="recent"/>
<parameter key="limit" value="%{tweets_to_search_through}"/>
<description align="center" color="transparent" colored="false" width="126">get all tweets mentioning the username</description>
</operator>
<operator activated="true" class="delay" compatibility="7.5.001" expanded="true" height="103" name="Delay" width="90" x="380" y="136">
<parameter key="delay" value="none"/>
<parameter key="delay_amount" value="1"/>
<description align="center" color="transparent" colored="false" width="126">make sure both information are in-place at the same time</description>
</operator>
<operator activated="true" class="filter_examples" compatibility="7.5.001" expanded="true" height="103" name="Filter Examples" width="90" x="514" y="136">
<list key="filters_list">
<parameter key="filters_entry_key" value="To-User-Id.eq.%{user-id}"/>
</list>
<description align="center" color="transparent" colored="false" width="126">select only tweets directed at our targeted user-id</description>
</operator>
<connect from_op="Get Twitter User Details" from_port="output" to_op="Extract Macro" to_port="example set"/>
<connect from_op="Extract Macro" from_port="example set" to_op="Delay" to_port="through 2"/>
<connect from_op="Search Twitter" from_port="output" to_op="Delay" to_port="through 1"/>
<connect from_op="Delay" from_port="through 1" to_op="Filter Examples" to_port="example set input"/>
<connect from_op="Filter Examples" from_port="example set output" to_port="output 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="source_input 2" spacing="0"/>
<portSpacing port="sink_output 1" spacing="0"/>
<portSpacing port="sink_output 2" spacing="0"/>
</process>
</operator>
<connect from_port="in 1" to_op="Documents to Data" to_port="documents 1"/>
<connect from_op="Documents to Data" from_port="example set" to_op="Loop Values" to_port="input 1"/>
<connect from_op="Loop Values" from_port="output 1" to_port="out 1"/>
<portSpacing port="source_in 1" spacing="0"/>
<portSpacing port="source_in 2" spacing="0"/>
<portSpacing port="source_in 3" spacing="0"/>
<portSpacing port="sink_out 1" spacing="0"/>
<portSpacing port="sink_out 2" spacing="0"/>
</process>
<description align="center" color="transparent" colored="false" width="126">Look through the given amount of tweets (via the set macro operator) to find tweets directed at the users listed in the provided document</description>
</operator>
<connect from_op="List of Users" from_port="output" to_op="Get Mentions from" to_port="in 1"/>
<connect from_op="Number of Tweets" from_port="through 1" to_op="Get Mentions from" to_port="in 2"/>
<connect from_op="Get Mentions from" from_port="out 1" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>

Cheers,

Philipp

Highlighted
RMStaff

Re: Search for replies to a particular twitter handle

[ Edited ]

Actually there is the option to use the "To-user-id" attribute to distinguish replies from retweets. It contains twitters internal id of the user that was mentioned. I created a small subprocess you can use to search for replies to a list of users.

Get Mentions for a list of Twitter UsersGet Mentions for a list of Twitter Users

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.5.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.5.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="text:create_document" compatibility="7.5.000" expanded="true" height="68" name="List of Users" width="90" x="112" y="34">
        <parameter key="text" value="rapidminer"/>
        <description align="center" color="transparent" colored="false" width="126">Provide a list of usernames as a document</description>
      </operator>
      <operator activated="true" class="set_macro" compatibility="7.5.001" expanded="true" height="82" name="Number of Tweets" width="90" x="112" y="187">
        <parameter key="macro" value="tweets_to_search_through"/>
        <parameter key="value" value="100"/>
        <description align="center" color="transparent" colored="false" width="126">set amount of tweets to scan through for mentions</description>
      </operator>
      <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Get Mentions from" width="90" x="313" y="34">
        <process expanded="true">
          <operator activated="true" class="text:documents_to_data" compatibility="7.5.000" expanded="true" height="82" name="Documents to Data" width="90" x="313" y="34">
            <parameter key="text_attribute" value="username"/>
          </operator>
          <operator activated="true" class="concurrency:loop_values" compatibility="7.5.001" expanded="true" height="82" name="Loop Values" width="90" x="514" y="34">
            <parameter key="attribute" value="username"/>
            <parameter key="iteration_macro" value="username"/>
            <process expanded="true">
              <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details" width="90" x="45" y="238">
                <parameter key="connection" value="Twitter Standard"/>
                <parameter key="user" value="%{username}"/>
                <description align="center" color="transparent" colored="false" width="126">get internal twitter id of the user for more accurate matching</description>
              </operator>
              <operator activated="true" class="extract_macro" compatibility="7.5.001" expanded="true" height="68" name="Extract Macro" width="90" x="179" y="238">
                <parameter key="macro" value="user-id"/>
                <parameter key="macro_type" value="data_value"/>
                <parameter key="attribute_name" value="Id"/>
                <parameter key="example_index" value="1"/>
                <list key="additional_macros"/>
                <description align="center" color="transparent" colored="false" width="126">make it available for future filtering</description>
              </operator>
              <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="45" y="34">
                <parameter key="connection" value="Twitter Standard"/>
                <parameter key="query" value="@%{username}"/>
                <parameter key="result_type" value="recent"/>
                <parameter key="limit" value="%{tweets_to_search_through}"/>
                <description align="center" color="transparent" colored="false" width="126">get all tweets mentioning the username</description>
              </operator>
              <operator activated="true" class="delay" compatibility="7.5.001" expanded="true" height="103" name="Delay" width="90" x="380" y="136">
                <parameter key="delay" value="none"/>
                <parameter key="delay_amount" value="1"/>
                <description align="center" color="transparent" colored="false" width="126">make sure both information are in-place at the same time</description>
              </operator>
              <operator activated="true" class="filter_examples" compatibility="7.5.001" expanded="true" height="103" name="Filter Examples" width="90" x="514" y="136">
                <list key="filters_list">
                  <parameter key="filters_entry_key" value="To-User-Id.eq.%{user-id}"/>
                </list>
                <description align="center" color="transparent" colored="false" width="126">select only tweets directed at our targeted user-id</description>
              </operator>
              <connect from_op="Get Twitter User Details" from_port="output" to_op="Extract Macro" to_port="example set"/>
              <connect from_op="Extract Macro" from_port="example set" to_op="Delay" to_port="through 2"/>
              <connect from_op="Search Twitter" from_port="output" to_op="Delay" to_port="through 1"/>
              <connect from_op="Delay" from_port="through 1" to_op="Filter Examples" to_port="example set input"/>
              <connect from_op="Filter Examples" from_port="example set output" to_port="output 1"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="source_input 2" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <connect from_port="in 1" to_op="Documents to Data" to_port="documents 1"/>
          <connect from_op="Documents to Data" from_port="example set" to_op="Loop Values" to_port="input 1"/>
          <connect from_op="Loop Values" from_port="output 1" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="source_in 3" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
        </process>
        <description align="center" color="transparent" colored="false" width="126">Look through the given amount of tweets (via the set macro operator) to find tweets directed at the users listed in the provided document</description>
      </operator>
      <connect from_op="List of Users" from_port="output" to_op="Get Mentions from" to_port="in 1"/>
      <connect from_op="Number of Tweets" from_port="through 1" to_op="Get Mentions from" to_port="in 2"/>
      <connect from_op="Get Mentions from" from_port="out 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

 

Cheers,

Philipp