Options

"Problem with R-script results"

ighybooighyboo Member Posts: 26 Contributor II
edited June 2019 in Help
Hi everyone,
I used the R extension before and always managed to get the data out using "outdata <-as.data.frame(Results)"

I'm currently working on a script that download twitter followers using the twitter API, everything work fine and I managed to create the results data-frame but Rapidminer doesn't like it and spit out this error:

"In order to import an R Data Frame as example set the data frame must provide attribute names"

I'm confused because when I check my result variable with colnames(outdata), the column names are assigned as you can see here:

colnames(Results)
[1] "id_str"            "name"              "screen_name"     
[4] "url"              "profile_image_url" "description"     
[7] "location"          "followers_count"  "friends_count"   
[10] "statuses_count"    "created_at" 



Anyone else had this problem before?

Here's the process and the script:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.015">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.3.015" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="generate_data_user_specification" compatibility="5.3.015" expanded="true" height="60" name="Generate Data by User Specification" width="90" x="45" y="30">
        <list key="attribute_values">
          <parameter key="target" value="&quot;WRADARltd&quot;"/>
        </list>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="r:execute_script_r" compatibility="5.3.000" expanded="true" height="76" name="Execute Script (R)" width="90" x="179" y="30">
        <parameter key="script" value="#Install libraries&#10;install.packages(&quot;httr&quot;);&#10;install.packages(&quot;rjson&quot;);&#10;install.packages(&quot;data.table&quot;);&#10;library(httr);&#10;library(rjson);&#10;library(data.table);&#10;&#10;&#10;#Declare API keys and inputs&#10;consumer_key = &quot;UansBjFHOm8fpkB4jx5mSiDTu&quot;;&#10;consumer_secret = &quot;dKTbQ0QKCqHaejnLyHOW0PHuOzpn5PIhjVmOkGMzcyLuaFrA7p&quot;;&#10;target = indata$target;&#10;&#10;&#10;#Auth&#10;secret &lt;- RCurl::base64(paste(consumer_key, consumer_secret, sep = &quot;:&quot;));&#10;req &lt;- POST(&quot;https://api.twitter.com/oauth2/token&quot;,&#10;  config(httpheader = c(&#10;    &quot;Authorization&quot; = paste(&quot;Basic&quot;, secret),&#10;    &quot;Content-Type&quot; = &quot;application/x-www-form-urlencoded;charset=UTF-8&quot;&#10;  )),&#10;  body = &quot;grant_type=client_credentials&quot;,&#10;  encode = &quot;multipart&quot;&#10;);&#10;&#10;&#10;#Extract the access token&#10;token &lt;- paste(&quot;Bearer&quot;, content(req)$access_token)&#10;&#10;&#10;#Request Followers count&#10;url &lt;- paste(&quot;https://api.twitter.com/1.1/users/lookup.json?screen_name=&quot;,target,sep=&quot;&quot;);&#10;req &lt;- GET(url, config(httpheader = c(&quot;Authorization&quot; = token)));&#10;json &lt;- content(req, as = &quot;text&quot;);&#10;tmp &lt;- fromJSON(json);&#10;num_followers &lt;- tmp[[1]]$followers_count;&#10;&#10;&#10;#If more than 5000 followers split ID request otherwise retrieve all in one request&#10;if (num_followers&lt;5000) {&#10;&#10;&#9;#Request &lt;5000 IDs &#10;&#9;url &lt;- paste(&quot;https://api.twitter.com/1.1/followers/ids.json?screen_name=&quot;,target,sep=&quot;&quot;);&#10;&#9;req &lt;- GET(url, config(httpheader = c(&quot;Authorization&quot; = token)));&#10;&#9;json &lt;- content(req, as = &quot;text&quot;);&#10;&#9;IDs &lt;- fromJSON(json);&#10;&#9;IDlist &lt;- data.frame(IDs$ids);&#10;&#9;&#10;&#9;} else {&#10;&#9;&#10;&#9;#Request batches of IDs&#10;&#9;batches &lt;- ceiling(num_followers/5000);&#10;&#9;batch_size &lt;- ceiling(num_followers/batches);&#10;&#9;curs &lt;- &quot;-1&quot;;&#10;&#9;IDlist &lt;- data.frame();&#10;&#9;for (i in 1:batches) {&#10;&#9;&#9;url &lt;- paste(&quot;https://api.twitter.com/1.1/followers/ids.json?cursor=&quot;,curs,&quot;&amp;screen_name=&quot;,target,&quot;&amp;count=&quot;,batch_size,sep=&quot;&quot;);&#10;&#9;&#9;req &lt;- GET(url, config(httpheader = c(&quot;Authorization&quot; = token)));&#10;&#9;&#9;json &lt;- content(req, as = &quot;text&quot;);&#10;&#9;&#9;IDs &lt;- fromJSON(json);&#10;&#9;&#9;tmp &lt;- data.frame(IDs$ids);&#10;&#9;&#9;IDlist &lt;- rbind(IDlist, tmp);&#10;&#9;&#9;curs &lt;- IDs$next_cursor_str;&#10;&#9;&#9;Sys.sleep(60);&#10;&#9;&#9;};&#10;&#9;};&#10;&#10;&#10;#Loop through IDs in batches to retrieve full profiles&#10;batches &lt;- ceiling(num_followers/100);&#10;batch_size &lt;- ceiling(num_followers/batches);&#10;BatchIDcounter &lt;- 0;&#10;FollowersList &lt;- data.frame();&#10;for (i in 1:batches) {&#10;&#9;start &lt;- BatchIDcounter;&#10;&#9;end &lt;- BatchIDcounter+batch_size;&#10;&#9;if (end&gt;length(IDlist[,1])) {&#10;&#9;&#9;end &lt;-length(IDlist[,1]);&#10;&#9;&#9;batch_size &lt;- (end-start);&#10;&#9;};&#10;&#9;BatchIDs &lt;- paste(IDlist[start:end,1],collapse=&quot;,&quot;);&#10;&#9;url &lt;- paste(&quot;https://api.twitter.com/1.1/users/lookup.json?include_entities=false&amp;user_id=&quot;,BatchIDs,sep=&quot;&quot;);&#10;&#9;req &lt;- GET(url, config(httpheader = c(&quot;Authorization&quot; = token)));&#10;&#9;json &lt;- content(req, as = &quot;text&quot;);&#10;&#9;profiles &lt;- fromJSON(json);&#10;&#9;for (j in 1:batch_size) {&#10;&#9;&#9;tmp &lt;- data.table(id_str=&quot;&quot;,name=&quot;&quot;,screen_name=&quot;&quot;,url=&quot;&quot;,profile_image_url=&quot;&quot;,description=&quot;&quot;,location=&quot;&quot;,followers_count=&quot;&quot;,friends_count=&quot;&quot;,statuses_count=&quot;&quot;,created_at=&quot;&quot;);&#10;&#9;&#9;if (length(profiles[]$id_str)==0) {tmp$id_str=&quot;&quot;} else{tmp$id_str=head(profiles[]$id_str[1])};&#10;&#9;&#9;if (length(profiles[]$name)==0) {tmp$name=&quot;&quot;} else{tmp$name=head(profiles[]$name[1])};&#10;&#9;&#9;if (length(profiles[]$screen_name)==0) {tmp$screen_name=&quot;&quot;} else{tmp$screen_name=head(profiles[]$screen_name[1])};&#10;&#9;&#9;if (length(profiles[]$url)==0) {tmp$url=&quot;&quot;} else{tmp$url=head(profiles[]$url[1])};&#10;&#9;&#9;if (length(profiles[]$profile_image_url)==0) {tmp$profile_image_url=&quot;&quot;} else{tmp$profile_image_url=head(profiles[]$profile_image_url[1])};&#10;&#9;&#9;if (length(profiles[]$description)==0) {tmp$description=&quot;&quot;} else{tmp$description=head(profiles[]$description[1])};&#10;&#9;&#9;if (length(profiles[]$location)==0) {tmp$location=&quot;&quot;} else{tmp$location=head(profiles[]$location[1])};&#10;&#9;&#9;if (length(profiles[]$url)==0) {tmp$url=&quot;&quot;} else{tmp$url=head(profiles[]$url[1])};&#10;&#9;&#9;if (length(profiles[]$followers_count)==0) {tmp$followers_count=&quot;&quot;} else{tmp$followers_count=head(profiles[]$followers_count[1])};&#10;&#9;&#9;if (length(profiles[]$friends_count)==0) {tmp$friends_count=&quot;&quot;} else{tmp$friends_count=head(profiles[]$friends_count[1])};&#10;&#9;&#9;if (length(profiles[]$statuses_count)==0) {tmp$statuses_count=&quot;&quot;} else{tmp$statuses_count=head(profiles[]$statuses_count[1])};&#10;&#9;&#9;if (length(profiles[]$created_at)==0) {tmp$created_at=&quot;&quot;} else{tmp$created_at=head(profiles[]$created_at[1])};&#10;&#9;&#9;FollowersList &lt;- rbind(FollowersList, tmp);&#10;&#9;};&#10;&#9;BatchIDcounter &lt;- end;&#10;&#9;Sys.sleep(5);&#10;};&#10;&#10;&#10;&#10;#Output&#10;outdata &lt;- as.data.frame(FollowersList);&#10;"/>
        <enumeration key="inputs">
          <parameter key="name_of_variable" value="indata"/>
        </enumeration>
        <list key="results">
          <parameter key="outdata" value="Data Table"/>
        </list>
      </operator>
      <operator activated="true" breakpoints="before" class="write_excel" compatibility="5.3.015" expanded="true" height="76" name="Write Excel" width="90" x="447" y="30">
        <parameter key="excel_file" value="C:\Documents and Settings\menghii1\Desktop\first.xlsx"/>
        <parameter key="file_format" value="xlsx"/>
      </operator>
      <connect from_op="Generate Data by User Specification" from_port="output" to_op="Execute Script (R)" to_port="input 1"/>
      <connect from_op="Execute Script (R)" from_port="output 1" to_op="Write Excel" to_port="input"/>
      <connect from_op="Write Excel" from_port="through" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>
Tagged:

Answers

  • Options
    ighybooighyboo Member Posts: 26 Contributor II
    P.S.

    Here the log details:
    SEVERE: Process failed: In order to import an R Data Frame as example set the data frame must provide attribute names.
    com.rapidminer.operator.UserError: In order to import an R Data Frame as example set the data frame must provide attribute names.
    at com.rapidminer.tools.r.translation.ExampleSetTranslator.getAttribute(ExampleSetTranslator.java:174)
    at com.rapidminer.tools.r.translation.ExampleSetTranslator.importObject(ExampleSetTranslator.java:105)
    at com.rapidminer.tools.r.translation.ExampleSetTranslator.importObject(ExampleSetTranslator.java:64)
    at com.rapidminer.operator.r.RScriptOperator.doWork(RScriptOperator.java:217)
    at com.rapidminer.operator.Operator.execute(Operator.java:866)
    at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
    at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:711)
    at com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:375)
    at com.rapidminer.operator.Operator.execute(Operator.java:866)
    at com.rapidminer.Process.run(Process.java:949)
    at com.rapidminer.Process.run(Process.java:873)
    at com.rapidminer.Process.run(Process.java:832)
    at com.rapidminer.Process.run(Process.java:827)
    at com.rapidminer.Process.run(Process.java:817)
    at com.rapidminer.gui.ProcessThread.run(ProcessThread.java:63)

  • Options
    ighybooighyboo Member Posts: 26 Contributor II
    Has anyone looked into this?

    I see that the same problem appeared in another post http://rapid-i.com/rapidforum/index.php/topic,7128.msg24720.html#msg24720.
    "tennenrishin" reported that this issue only happened when NA values were present however I tested that and in my case it happens even if all the values are present.
    I even tried to explicitly declare every column of the DataFrame when creating it but no luck...

    This was actually one of the main reasons why I stopped using RM and moved to do most of my work directly in R... But that's a real pity as there are many things that I like about RM.

    Any possible solution?
  • Options
    javemarjavemar Member Posts: 23 Contributor II
    Hello


    I had the same problem. What I figured is when you put data from R to Rapidminer you must assure that all character data is a factor. Example


    If you have a input variable name as "input" and you want to create a column "newcolumn" with the value  "newvalue"  may you type

    input$newcolumn<-"newvalue"

    But this expression will give you a error.

    However, if you try

    input$newcolumn<-as.factor("newvalue" )

    Rapidminer will recognize the new attribute










  • Options
    David_ADavid_A Administrator, Moderator, Employee, RMResearcher, Member Posts: 297 RM Research
    As a general tip, there is the new R scripting extension, which offers a much smoother integration of R for RapidMiner:
    https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_r_scripting
  • Options
    gbortz27gbortz27 Member Posts: 22 Contributor II
    I installed the new extender and dont see the perspective load on restarting Rapidminer..it does state when I tried to install the extention , that the extention is already installed , any ideas ?

    Thanks
Sign In or Register to comment.