"Twitter API error for multiple user details extraction"

m_okem_oke Member Posts: 11 Contributor I
edited June 2019 in Help
<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="179" y="85">
<parameter key="connection" value="TwitterConnection1"/>
<parameter key="query" value="&quot;rapidminer&quot; -rt -http"/>
<parameter key="limit" value="20"/>
<parameter key="language" value="en"/>
</operator>
<operator activated="true" class="filter_examples" compatibility="7.6.001" expanded="true" height="103" name="Filter Examples" width="90" x="447" y="85">
<list key="filters_list">
<parameter key="filters_entry_key" value="Retweet-Count.ge.1"/>
</list>
</operator>
<operator activated="true" breakpoints="after" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes" width="90" x="648" y="85">
<parameter key="attribute_filter_type" value="single"/>
<parameter key="attribute" value="From-User"/>
</operator>
<operator activated="true" breakpoints="after" class="concurrency:loop_values" compatibility="7.6.001" expanded="true" height="82" name="Loop Values (2)" width="90" x="849" y="85">
<parameter key="attribute" value="From-User"/>
<process expanded="true">
<operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details (2)" width="90" x="514" y="34">
<parameter key="connection" value="NewConnection"/>
<parameter key="user" value="%{loop_value}"/>
</operator>
<connect from_op="Get Twitter User Details (2)" from_port="output" to_port="output 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="source_input 2" spacing="0"/>
<portSpacing port="sink_output 1" spacing="0"/>
<portSpacing port="sink_output 2" spacing="0"/>
</process>
</operator>
<operator activated="true" class="append" compatibility="7.6.001" expanded="true" height="82" name="Append" width="90" x="1050" y="85"/>
<connect from_op="Search Twitter" from_port="output" to_op="Filter Examples" to_port="example set input"/>
<connect from_op="Filter Examples" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
<connect from_op="Select Attributes" from_port="example set output" to_op="Loop Values (2)" to_port="input 1"/>
<connect from_op="Loop Values (2)" from_port="output 1" to_op="Append" to_port="example set 1"/>
<connect from_op="Append" from_port="merged set" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>

I built an experiment in RapidMiner that does the following:

1) Using the 'Search Twitter', it searches twitter for keywords/queries

2) Using 'Filter keywords', it filters the results by number of retweets greater than 1

3) Using 'Selelect Attributes', it selects only the attribute 'From-User'

4) Using 'Loop Values' with emebedded 'Get Twitter user details', I extracted the user details of all users in (3) above

5) Using the 'Append' tool, I appended all the user details into 1

 

On running the experiment, it gave an error of 'Twitter API error' at the 'Get Twitter User Details' tool.

 

Not sure how to fix it, any help will be apreciated.

 

XML code of the experiment is attached:

 

Best Answer

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist
    Solution Accepted

    Hi @m_oke,

     

    the following line in your log file explains the problem:

     

    Caused by: Server returned HTTP response code: 400 for URL: https://api.twitter.com/1.1/users/show.json?screen_name=Rebecca Lopez&include_entities=true

    The Operator "Get Twitter User Details" with the parameter name searches for the Screen name of a user.

    That is the one with @. Those do not have blanks. If you can obtain those names you can use them.

    Otherwise you can use the parameter id within "Get Twitter User Details". The id is a number and is also available from the Operator Search Twitter (Attribute: From-User-Id).

     

    An example process can be found in this community post.

     

    Best regards,

    Edin

     

Answers

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    HI @m_oke,

     

    can you please provide the file rapidminer-studio.log which is located in the directory <userhome>/.RapidMiner/

     

    Thanks,

    Edin

  • m_okem_oke Member Posts: 11 Contributor I

    Hi @Edin_Klapic

    See the log below:

     

    Nov 13, 2017 10:48:44 AM com.rapidminer.tools.ResultService init
    INFO: No filename given for result file, using stdout for logging results!
    Nov 13, 2017 10:48:44 AM com.rapidminer.Process run
    INFO: Process starts
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread run
    SEVERE: Process failed: Error on connecting to Twitter API.
    com.rapidminer.operator.UserError: Error on connecting to Twitter API.
        at com.rapidminer.operator.web.services.twitter.GetTwitterUserDetailsOperator.createExampleSet(GetTwitterUserDetailsOperator.java:218)
        at com.rapidminer.operator.io.AbstractExampleSource.read(AbstractExampleSource.java:49)
        at com.rapidminer.operator.io.AbstractExampleSource.read(AbstractExampleSource.java:33)
        at com.rapidminer.operator.io.AbstractReader.doWork(AbstractReader.java:126)
        at com.rapidminer.operator.Operator.execute(Operator.java:1004)
        at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:77)
        at com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:812)
        at com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:807)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:807)
        at com.rapidminer.extension.concurrency.operator.process_control.loops.AbstractLoopOperator.doIteration(AbstractLoopOperator.java:304)
        at com.rapidminer.extension.concurrency.operator.process_control.loops.AbstractLoopOperator$1.call(AbstractLoopOperator.java:191)
        at com.rapidminer.extension.concurrency.operator.process_control.loops.AbstractLoopOperator$1.call(AbstractLoopOperator.java:182)
        at com.rapidminer.extension.concurrency.execution.BackgroundExecutionService$ExecutionCallable.call(BackgroundExecutionService.java:357)
        at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
    Caused by: Server returned HTTP response code: 400 for URL: https://api.twitter.com/1.1/users/show.json?screen_name=Rebecca Lopez&include_entities=true
    Relevant discussions can be found on the Internet at:
        http://www.google.co.jp/search?q=381aea58 or
        http://www.google.co.jp/search?q=3393bbb1
    TwitterException{exceptionCode=[381aea58-3393bbb1 720b1cc8-3ee3b645 720b1cc8-3ee3b636], statusCode=-1, message=null, code=-1, retryAfter=-1, rateLimitStatus=null, version=4.0.1}
        at twitter4j.HttpClientImpl.handleRequest(HttpClientImpl.java:179)
        at twitter4j.HttpClientBase.request(HttpClientBase.java:53)
        at twitter4j.HttpClientBase.get(HttpClientBase.java:71)
        at twitter4j.TwitterImpl.get(TwitterImpl.java:1950)
        at twitter4j.TwitterImpl.showUser(TwitterImpl.java:874)
        at com.rapidminer.operator.web.services.twitter.GetTwitterUserDetailsOperator.createExampleSet(GetTwitterUserDetailsOperator.java:168)
        ... 18 more
    Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: https://api.twitter.com/1.1/users/show.json?screen_name=Rebecca Lopez&include_entities=true
        at sun.reflect.GeneratedConstructorAccessor328.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
        at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:91)
        at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1466)
        at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1464)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1463)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
        at twitter4j.HttpResponseImpl.<init>(HttpResponseImpl.java:50)
        at twitter4j.HttpClientImpl.handleRequest(HttpClientImpl.java:143)
        ... 23 more
    Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: https://api.twitter.com/1.1/users/show.json?screen_name=Rebecca Lopez&include_entities=true
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1876)
        at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:91)
        at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1466)
        at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1464)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1463)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
        at twitter4j.HttpResponseImpl.<init>(HttpResponseImpl.java:35)
        ... 24 more

    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE: Here:
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE:           Process[1] (Process)
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE:            subprocess 'Main Process'
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE:              +- Search Twitter[1] (Search Twitter)
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE:              +- Filter Examples[1] (Filter Examples)
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE:              +- Select Attributes[1] (Select Attributes)
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE:        ==>   +- Loop Values (2)[1] (Loop Values)
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE:            subprocess 'LoopValues'
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE:              |     +- Get Twitter User Details (2)[4] (Get Twitter User Details)
    Nov 13, 2017 10:48:45 AM com.rapidminer.gui.ProcessThread logProcessTreeList
    SEVERE:              +- Append[0] (Append)
    Nov 13, 2017 10:48:46 AM com.rapidminer.operator.Operator stop
    INFO: Loop Values (2): Process stopped.
    Nov 13, 2017 10:48:46 AM com.rapidminer.operator.Operator stop
    INFO: Loop Values (2): Process stopped.

Sign In or Register to comment.