Highest Rated Posts

SGolbert

March 8

Re: Combining RapidMiner with Python and R

Hi @mlg1988 in my opinion R and Python help cover some niche cases which RapidMiner not yet covers, or does it poorly (e.g.: the web mining extension). If you work with the RapidMiner platform, you don't really need to program much, only a few lines to get some special functionality, via the scripting operators. The project framework that RM gives you is unparalleled in the R and Python world. There are some cases when some tasks are easier to do with a script than in RapidMiner, in those cases I would insist with RM for a given time, because once you solve a problem in RM, the future use of the process (changes, debugging, operationalization) is better if you don't depend on external tools. Regards, Sebastian

sgenzer

March 4

Cross-Validation and its outputs in RM Studio

We get a lot of questions about how exactly the Cross Validation operator works, and what those 'exa', 'mod', and 'tes' outputs produce, in RM Studio. Hence it seems a good idea to explain this in depth as a KB article. So here I go... What is Cross Validation? Cross Validation is a method used to find a true estimate of the performance of mathematical models built on relatively small data sets. It builds a series of models on subsets of the data and tests each model on the remainder of the data to determine an average performance metric of the models, rather than one performance metric of one model. Split Validation, a much simpler process where you just split the data once into a 'training' and 'testing' set, works very well for very large data sets. But it's Achilles heel is its assumption that the testing set is a representative sample of the entire data set. This is likely true for large data sets, but less and less likely the smaller your data set is.  How does Cross Validation work? Cross Validation works by dividing the data set first into n partitions, and then building n unique models one at a time (each iteration is called a "fold"). For each iteration, a model is created using n-1 partitions and using the nth partition. The model itself is not relevant; what is retained is the model performance. This is repeated using a different partition as the hold-out set until each partition is used. After the n performances are calculated, we calculate the average performance (and its standard deviation) of all n models built during the cross validation. Lastly, we build a true model with the entire data set and display this calculated average performance. Very Simplistic Example of Cross Validation Let's walk through exactly how it works using a very simple example. Here is a data set where we have one independent variable (x) and one dependent variable (y). The goal is to produce the best linear regression model of these data using cross validation. The y values were randomly generated. In this example we're going to use linear sampling, 5-fold cross validation, and RSME as our performance metric: * Linear sampling - we will pull our hold-out sets in order which the data sets is given to us * 5-fold Cross Validation - we will perform five iterations to find five performances using five hold-out sets * RSME - the root-mean-squared-error of the predicted values of each hold-out set vs the actual values Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Hence: average RSME =  avg (0.347, 0.286, 0.306, 0.289, 0.321) = 0.310  standard deviation of RSME = stdev (0.347, 0.286, 0.306, 0.289, 0.321) = 0.025 The model representing this performance metric is none of the models shown above; it is a new model produced on the entire data set. Sometimes this is considered the "n+1" th iteration. Calculate the actual model What are the outputs of the Cross Validation operator in RapidMiner Studio? Ok now we can answer this question! They are as follows: * mod - the actual mathematical model built on the entire data set (see last step above). Usually only the relevant coefficients or equivalent are displayed. * exa - the original ExampleSet (data). This is simple a pass-thru. * tes - the testing ExampleSet. Note that this is NOT the application of the final model on the data set; rather it displays all the hold-out testing sets from the n-fold cross validation in one appended list. * per = the average performance of the n-fold cross validation For our example, these are as follows: Why should we go through all this trouble? I am going to just insert a few paragraphs from founder @IngoRM here: (Quote) Hope this helps some people. Thank you @yyhuang and @IngoRM for your help!  Scott

kypexin

March 1

Re: how to select important variables for my model

Hi @ed1305 New AUTOMATIC FEATURE ENGINEERING operator does a great job here, that's the one I use every time I need to pick the best features out of original dataset. 

sgenzer

February 26

How to Become a Unicorn

Who are those "Unicorns" on the RapidMiner Community? If you've been wondering about this question, let me explain how the rankings work here and how you too can achieve this level of distinction! For starters, in Data Science, a "Unicorn" is someone who's highly talented and can use DS tools with competence to solve some of the world's hardest problems. They are very few and far between - hence the term "Unicorn" to describe a rare animal to find.  In the RapidMiner Community, a "Unicorn" is someone who has proved their proficiency by providing meaningful answers and solutions to people's questions or by becoming certified in RapidMiner Studio. The gamification system works on only three metrics: * # of days that you have been a member * # of new discussions/comments (posts) you have made * # of "points" you have earned The first two should be pretty obvious; the points scheme may need some explaining. In short, you earn points here for actions that benefit the community. They are: * Receiving a "Like" or "Up Vote" from a discussion/comment you made = 1 point * Receiving a "Promote" from a moderator = 10 points * Receiving a "Badge" = 1 to 100 points depending on the badge * Answering a Question = 10 points * Having an answer be marked as a Solution = 40 points * Becoming a RapidMiner Certified Analyst = 100 points * Becoming a RapidMiner Certified Expert = 200 points So how do you move up in the rankings? * "Newbie" - assigned as soon as you sign in * "Learner I" - member for 2 days; earned 5 points * "Learner II" - member for 4 days; earned 10 points * "Learner III" - member for 6 days; earned 15 points * "Contributor I" - member for 2 days; earned 10 points; made 1 post * "Contributor II" - member for 6 days; earned 40 points; made 5 posts * "Maven" - member for 7 days; earned 50 points; made 15 posts * "Guru" - member for 30 days; earned 60 points; made 30 posts * "Unicorn" - member for 30 days; earned 200 points; made 40 posts So there you go - no secrets here. So go ahead and start earning those points!! Scott

BalazsBarany

February 25

Powerful JSON processing with jq

jq is a command line tool for processing JSON documents. It can filter, transform and restructure documents to format them in the way we want. The JSON documents we have to work with are becoming more and more complex. Web APIs often generate documents with hierarchic structure and optional elements.  Data mining, however, needs a tabular structure, without hierarchic elements, and if possible without missing data. jq helps us with the transformation of relevant parts of input documents into this shape. Consider the following example document: {     "count": 3,     "category": "example",     "elements": [         {             "id": 1,             "description": "first element",             "tags": ["tag1"]         },         {             "id": 2,             "description": "second element",             "optional": "optional element",             "tags": []         },         {             "id": 3,             "description": "third element",             "tags": ["tag1", "tag2"]         }     ] } This shows the usual pitfalls of complex JSON documents: * Elements on different hierarchy levels: category, elements/id etc. * Optional elements: elements[2]/optional * Variable number of elements: elements/tags It is really hard to process this kind of document in RapidMiner. If you install the Web Mining extension, you get JSON to XML; in Text Processing there's JSON to Data. However, JSON to XML tends to fail on more complex documents, and JSON to Data produces an example set with hundreds or thousands of attributes that might be complex or impossible to process with a series of Transpose, Pivot and similar operators. So one might want to turn to jq. It can be used as a command line utility to process data. But the easiest way to try it is online at jqplay.org. Let's create a table with the category, the element id and the tags. The jq expression for this is: {count, category, elements: .elements[] } | {category, id: .elements.id, tag: .elements.tags[]} Scary for sure in the first moment! But when you look at it, it’s built of simple elements. You can always execute it step by step at jqplay to see the effects of each transformation. In the first step (the steps being delimited by the pipe symbol „|“) we declare the elements we want to process. We build an object list with { }, taking count and category from the top level and an array of the elements. count and category are repeated to create a proper table. In the second step we select category and the object id-s, which were on different levels previously. The tags are selected as an array. Using the syntax name: .element.element we can select elements and name them. The result of this step is a list of objects having category, id and tag in a table, suitable for writing into a relational database or processing in a data mining tool. Using jq with RapidMiner It would be useful to process these kinds of documents with jq in RapidMiner. This is what I did, using jackson-jq, a Java implementation of jq. A zip file is attached to this post, it contains the necessary libraries and two example processes.  To prepare, we need to copy the jackson-jq jar file and two dependencies into the RapidMiner Studio lib directory. Then we’re able to use the jq functionality in the built-in Groovy scripting operator (Execute Script). I created two RapidMiner processes to make the application easier. These can be used in other processes. There is one variant working on tables (example sets), here you specify the name of the input attribute containing your documents and the target attribute for the transformation result. The other variant works on Document objects, like those coming from Get Page. In both cases you specify the jq expression and set up the output options. You can indent the output, and convert the result to CSV. The CSV formatted result can be easily transformed to an example set — this is a frequent use case. If you want to see CSV output in jqplay, check „Raw Output“ and append the following: | [ .category, .id, .tag ] | @csv This creates an array (with the [] syntax) and lists the elements in the output. The result is converted with the @csv step. (The RapidMiner process does this automatically if the csv output is selected.) This, together with some practicing in jqplay, enables processes that can transform complex JSON documents to straight tables. To process different parts and structures in the document, just multiply it and apply different jq expressions on the copies.

rfuentealba

February 24

Re: Updating Anaconda and Tensorflow breaks the execute Python extension.

Hi @hughesfleming68, I'm running on 3.7.2 without problems. When you install a new Python distribution with conda update, you create a bare one as if you would have installed Miniconda. The problem isn't solved by downgrading but by installing Pandas on the new version after upgrading. What I normally do: * Install Anaconda. * Create an environment solely for RapidMiner: $ conda create --name=rapidminer python=3.7  * Activate the environment for RapidMiner: $ conda activate rapidminer * Install pandas for the new environment: $ conda install pandas * List the environments I have: $ conda env list * Copy the path of the environment I created, add /bin/python to the end and use that path to configure the Python extension. * Deactivate the environment: $ conda deactivate That way, nothing of what I do with other environments affect the one I use for RapidMiner. Hope this helps, Rodrigo.

Latest Posts

RapidMiner Community

Discussion List

• New RapidMiner research project - dealing with real-time, interactive extreme-scale analytics

  Announcement 84 views 2 comments 7 points Most recent by David_A March 5Help
• problem with result of ICA ?

  Responded But No Solution 13 views 3 comments 0 points Most recent by varunm1 5:36PMHelp
• is there any additional information can I got it from rapidminer cloud?

  Responded But No Solution 39 views 4 comments 0 points Most recent by norah 3:27PMHelp
• W-M5P

  Responded But No Solution 13 views 1 comment 0 points Most recent by varunm1 3:17PMHelp
• H20 Deep Learning error on the first run.

  Responded But No Solution 34 views 3 comments 0 points Most recent by hughesfleming68 1:01PMHelp
• Replacing numeric values (Map) in Turbo Prep

  Responded But No Solution 22 views 2 comments 0 points Most recent by IngoRM 12:15PMHelp
• Time series with multiple attributes

  Responded But No Solution 28 views 1 comment 0 points Most recent by hughesfleming68 10:05AMHelp
• Plot

  Responded But No Solution 16 views 2 comments 0 points Most recent by varunm1 10:04AMHelp
• how to tokenize within document to data rapidminer

  Responded But No Solution 24 views 1 comment 0 points Most recent by Prentice 6:32AMHelp
• Why is Scatter 3D plot in RM 9.2 limited to 2000 rows????

  Responded But No Solution 19 views 1 comment 0 points Most recent by Marco_Boeck 5:58AMHelp
• How can i know which attribute is PC1 in PCA

  Solution Accepted 41 views 5 comments 0 points Most recent by David_A 4:30AMHelp
• List Sharepoint Files

  Responded But No Solution 24 views 1 comment 0 points Most recent by varunm1 12:46AMHelp
• Import data

  Responded But No Solution 17 views 1 comment 0 points Most recent by hughesfleming68 March 24Help
• i can't visualize the result. It shows "could not initialize HTML5 browser! See log for details".

  Responded But No Solution 25 views 5 comments 0 points Most recent by azleena March 24Help
• PM SEVERE: Failed to create HTML5 browser!

  Responded But No Solution 35 views 2 comments 0 points Most recent by azleena March 23Help
• How to extract YEAR from a string?

  Responded But No Solution 21 views 2 comments 0 points Most recent by Telcontar120 March 23Help
• สวัสดีครับ แนะนำตัวกันครับ :)

  Responded But No Solution 319 views 16 comments 5 points Most recent by chimdolin March 23Help
• Twitter operators and their rate limits

  17 views 2 comments 0 points Most recent by sgenzer March 22Knowledge Base
• Execute script Operator + include many .java files

  Responded But No Solution 23 views 1 comment 0 points Most recent by mschmitz March 22Help
• Prediction

  Responded But No Solution 25 views 1 comment 0 points Most recent by varunm1 March 22Help
• AttributeTypeException, Keras changes type of label to rm_metadata("nominal", prediction")

  Responded But No Solution 32 views 1 comment 0 points Most recent by hughesfleming68 March 22Help
• Applying Machine Learning to Text Mining with Amazon S3 and RapidMiner

  19 views 3 comments 1 point Most recent by sgenzer March 22Knowledge Base
• "Process Documents from Data" and "Generate TF-IDF"

  Solution Accepted 15 views 2 comments 0 points Most recent by Prentice March 22Help
• Rapid Miner

  Responded But No Solution 18 views 1 comment 0 points Most recent by David_A March 22Help
• How to turn a colored histogram into a stacked bar chart?

  Solution Accepted 38 views 5 comments 0 points Most recent by sgenzer March 21Help
• How to invert the example order?

  Solution Accepted 19 views 2 comments 0 points Most recent by Newbie March 21Help
• 100,000 rows of data limit with an Educational license.

  Responded But No Solution 66 views 2 comments 1 point Most recent by p15248633 March 21Help
• Writing into multiple sheets in an Excel

  Solution Accepted 38 views 6 comments 0 points Most recent by varunm1 March 21Help
• Scoring - Using Classification models

  Responded But No Solution 44 views 6 comments 0 points Most recent by IngoRM March 21Help
• how to select important variables for my model

  Solution Accepted 34 views 3 comments 0 points Most recent by Brian_Wells March 21Help
• Process Documents from Data error

  Responded But No Solution 36 views 2 comments 0 points Most recent by Prentice March 21Help