Collect Twitter user data automatically with the Get Twitter User Details Operator

julian_d

Hi there,

First of all, I am RapidMiner rooky – my apologies for any stupid question in advance. I have not found an answer to my question in the community, so here I go:

I would like to analyse Tweets containing keywords using the “Search Twitter” Operator and then automatically lookup the users that posted something containing the keywords. I know the “Get Twitter User Details” Operator allows me to fill in user-ids manually (one at a time) but I would like RapidMiner to take the user-id from the “Search Twitter” Operator and do it automatically. In the Youtube Tutorial “Discover Twitter Content Using RapidMiner” (https://www.youtube.com/watch?v=ia2iV5Ws3zo), posted in another query, I learnt that a Macro might be a way to do it but I have not made it yet to read the user-ids of the “Search Twitter” results with the Macro and then pass it over to the “Get Twitter User Details” Operator. I recon, I somehow need a loop to do it. I do not expect a final, clean solution but maybe you guys have a hint. The xml code should be attached for better understanding of my problem and ideas.

Thanks very much in advance!

Julian

 

<?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
  <operator activated="true" class="social_media:search_twitter" compatibility="8.1.000" expanded="true" height="68" name="Search Twitter" width="90" x="45" y="85">
    <parameter key="connection" value="Twitter_DUJ"/>
    <parameter key="query" value="Energiewende"/>
    <parameter key="result_type" value="recent or popular"/>
    <parameter key="limit" value="100"/>
    <parameter key="filter_by_geo_location" value="false"/>
    <parameter key="radius_unit" value="miles"/>
  </operator>
</process>
<?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
  <operator activated="true" class="select_attributes" compatibility="8.1.001" expanded="true" height="82" name="Select Attributes" width="90" x="179" y="85">
    <parameter key="attribute_filter_type" value="single"/>
    <parameter key="attribute" value="From-User-Id"/>
    <parameter key="attributes" value=""/>
    <parameter key="use_except_expression" value="false"/>
    <parameter key="value_type" value="attribute_value"/>
    <parameter key="use_value_type_exception" value="false"/>
    <parameter key="except_value_type" value="time"/>
    <parameter key="block_type" value="attribute_block"/>
    <parameter key="use_block_type_exception" value="false"/>
    <parameter key="except_block_type" value="value_matrix_row_start"/>
    <parameter key="invert_selection" value="false"/>
    <parameter key="include_special_attributes" value="true"/>
  </operator>
</process>
<?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
  <operator activated="true" class="set_macros" compatibility="8.1.001" expanded="true" height="124" name="Set Macros" width="90" x="313" y="85">
    <list key="macros">
      <parameter key="userid" value="#keyword1"/>
    </list>
  </operator>
</process>
<?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
  <operator activated="true" class="social_media:get_twitter_user_details" compatibility="8.1.000" expanded="true" height="68" name="Get Twitter User Details" width="90" x="380" y="391">
    <parameter key="connection" value="Twitter_DUJ"/>
    <parameter key="query_type" value="name"/>
    <parameter key="user" value="%{keyword1}"/>
  </operator>
</process>
<?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
  <operator activated="true" class="select_attributes" compatibility="8.1.001" expanded="true" height="82" name="Select Attributes (2)" width="90" x="514" y="391">
    <parameter key="attribute_filter_type" value="subset"/>
    <parameter key="attribute" value="From-User"/>
    <parameter key="attributes" value="Id|Language|Location|Name|Time-Zone|Tweets"/>
    <parameter key="use_except_expression" value="false"/>
    <parameter key="value_type" value="attribute_value"/>
    <parameter key="use_value_type_exception" value="false"/>
    <parameter key="except_value_type" value="time"/>
    <parameter key="block_type" value="attribute_block"/>
    <parameter key="use_block_type_exception" value="false"/>
    <parameter key="except_block_type" value="value_matrix_row_start"/>
    <parameter key="invert_selection" value="false"/>
    <parameter key="include_special_attributes" value="false"/>
  </operator>
</process>

 

Thomas_Ott

@julian_d also have a looksee here: https://community.rapidminer.com/t5/Product-Feedback/HELP-obtaining-Twitter-user-details-I-m-new-to-RapidMiner/idi-p/38748#M26602

 

There's a sample process that might just be what you're looking for. 

Thomas_Ott

@julian_d the XML you posted is corrupted, you'll need to open the XML view and copy and paste it into the </> here. 

 

I'm the creator of that video and I've done what you're looking to do with moderate success due to the different ranges of user-ids and a strange bug in the Twitter operators. It's rumored to be fixed and I'd have to dig around for the old process. 

julian_d

Thanks a lot @Thomas_Ott!! - A shame I had not found that one on my own.

 

In case somebody else is facing this issue too: It took me some time to realise that the Twitter Operators output change their attribute type from one RM version to another. So in case the attribute types do not match you might want to have a look at the Twitter Operator Parameters and adapt "Change compatibility".

 

Thanks again! Cheers

Thomas_Ott

@julian_d the community is your friend, it's filled with all kinds of good stuff. Good luck!