Highest Rated Posts

SGolbert

March 8

Re: Combining RapidMiner with Python and R

Hi @mlg1988 in my opinion R and Python help cover some niche cases which RapidMiner not yet covers, or does it poorly (e.g.: the web mining extension). If you work with the RapidMiner platform, you don't really need to program much, only a few lines to get some special functionality, via the scripting operators. The project framework that RM gives you is unparalleled in the R and Python world. There are some cases when some tasks are easier to do with a script than in RapidMiner, in those cases I would insist with RM for a given time, because once you solve a problem in RM, the future use of the process (changes, debugging, operationalization) is better if you don't depend on external tools. Regards, Sebastian

sgenzer

March 4

Cross-Validation and its outputs in RM Studio

We get a lot of questions about how exactly the Cross Validation operator works, and what those 'exa', 'mod', and 'tes' outputs produce, in RM Studio. Hence it seems a good idea to explain this in depth as a KB article. So here I go... What is Cross Validation? Cross Validation is a method used to find a true estimate of the performance of mathematical models built on relatively small data sets. It builds a series of models on subsets of the data and tests each model on the remainder of the data to determine an average performance metric of the models, rather than one performance metric of one model. Split Validation, a much simpler process where you just split the data once into a 'training' and 'testing' set, works very well for very large data sets. But it's Achilles heel is its assumption that the testing set is a representative sample of the entire data set. This is likely true for large data sets, but less and less likely the smaller your data set is.  How does Cross Validation work? Cross Validation works by dividing the data set first into n partitions, and then building n unique models one at a time (each iteration is called a "fold"). For each iteration, a model is created using n-1 partitions and using the nth partition. The model itself is not relevant; what is retained is the model performance. This is repeated using a different partition as the hold-out set until each partition is used. After the n performances are calculated, we calculate the average performance (and its standard deviation) of all n models built during the cross validation. Lastly, we build a true model with the entire data set and display this calculated average performance. Very Simplistic Example of Cross Validation Let's walk through exactly how it works using a very simple example. Here is a data set where we have one independent variable (x) and one dependent variable (y). The goal is to produce the best linear regression model of these data using cross validation. The y values were randomly generated. In this example we're going to use linear sampling, 5-fold cross validation, and RSME as our performance metric: * Linear sampling - we will pull our hold-out sets in order which the data sets is given to us * 5-fold Cross Validation - we will perform five iterations to find five performances using five hold-out sets * RSME - the root-mean-squared-error of the predicted values of each hold-out set vs the actual values Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Hence: average RSME =  avg (0.347, 0.286, 0.306, 0.289, 0.321) = 0.310  standard deviation of RSME = stdev (0.347, 0.286, 0.306, 0.289, 0.321) = 0.025 The model representing this performance metric is none of the models shown above; it is a new model produced on the entire data set. Sometimes this is considered the "n+1" th iteration. Calculate the actual model What are the outputs of the Cross Validation operator in RapidMiner Studio? Ok now we can answer this question! They are as follows: * mod - the actual mathematical model built on the entire data set (see last step above). Usually only the relevant coefficients or equivalent are displayed. * exa - the original ExampleSet (data). This is simple a pass-thru. * tes - the testing ExampleSet. Note that this is NOT the application of the final model on the data set; rather it displays all the hold-out testing sets from the n-fold cross validation in one appended list. * per = the average performance of the n-fold cross validation For our example, these are as follows: Why should we go through all this trouble? I am going to just insert a few paragraphs from founder @IngoRM here: (Quote) Hope this helps some people. Thank you @yyhuang and @IngoRM for your help!  Scott

kypexin

March 1

Re: how to select important variables for my model

Hi @ed1305 New AUTOMATIC FEATURE ENGINEERING operator does a great job here, that's the one I use every time I need to pick the best features out of original dataset. 

sgenzer

February 26

How to Become a Unicorn

Who are those "Unicorns" on the RapidMiner Community? If you've been wondering about this question, let me explain how the rankings work here and how you too can achieve this level of distinction! For starters, in Data Science, a "Unicorn" is someone who's highly talented and can use DS tools with competence to solve some of the world's hardest problems. They are very few and far between - hence the term "Unicorn" to describe a rare animal to find.  In the RapidMiner Community, a "Unicorn" is someone who has proved their proficiency by providing meaningful answers and solutions to people's questions or by becoming certified in RapidMiner Studio. The gamification system works on only three metrics: * # of days that you have been a member * # of new discussions/comments (posts) you have made * # of "points" you have earned The first two should be pretty obvious; the points scheme may need some explaining. In short, you earn points here for actions that benefit the community. They are: * Receiving a "Like" or "Up Vote" from a discussion/comment you made = 1 point * Receiving a "Promote" from a moderator = 10 points * Receiving a "Badge" = 1 to 100 points depending on the badge * Answering a Question = 10 points * Having an answer be marked as a Solution = 40 points * Becoming a RapidMiner Certified Analyst = 100 points * Becoming a RapidMiner Certified Expert = 200 points So how do you move up in the rankings? * "Newbie" - assigned as soon as you sign in * "Learner I" - member for 2 days; earned 5 points * "Learner II" - member for 4 days; earned 10 points * "Learner III" - member for 6 days; earned 15 points * "Contributor I" - member for 2 days; earned 10 points; made 1 post * "Contributor II" - member for 6 days; earned 40 points; made 5 posts * "Maven" - member for 7 days; earned 50 points; made 15 posts * "Guru" - member for 30 days; earned 60 points; made 30 posts * "Unicorn" - member for 30 days; earned 200 points; made 40 posts So there you go - no secrets here. So go ahead and start earning those points!! Scott

BalazsBarany

February 25

Powerful JSON processing with jq

jq is a command line tool for processing JSON documents. It can filter, transform and restructure documents to format them in the way we want. The JSON documents we have to work with are becoming more and more complex. Web APIs often generate documents with hierarchic structure and optional elements.  Data mining, however, needs a tabular structure, without hierarchic elements, and if possible without missing data. jq helps us with the transformation of relevant parts of input documents into this shape. Consider the following example document: {     "count": 3,     "category": "example",     "elements": [         {             "id": 1,             "description": "first element",             "tags": ["tag1"]         },         {             "id": 2,             "description": "second element",             "optional": "optional element",             "tags": []         },         {             "id": 3,             "description": "third element",             "tags": ["tag1", "tag2"]         }     ] } This shows the usual pitfalls of complex JSON documents: * Elements on different hierarchy levels: category, elements/id etc. * Optional elements: elements[2]/optional * Variable number of elements: elements/tags It is really hard to process this kind of document in RapidMiner. If you install the Web Mining extension, you get JSON to XML; in Text Processing there's JSON to Data. However, JSON to XML tends to fail on more complex documents, and JSON to Data produces an example set with hundreds or thousands of attributes that might be complex or impossible to process with a series of Transpose, Pivot and similar operators. So one might want to turn to jq. It can be used as a command line utility to process data. But the easiest way to try it is online at jqplay.org. Let's create a table with the category, the element id and the tags. The jq expression for this is: {count, category, elements: .elements[] } | {category, id: .elements.id, tag: .elements.tags[]} Scary for sure in the first moment! But when you look at it, it’s built of simple elements. You can always execute it step by step at jqplay to see the effects of each transformation. In the first step (the steps being delimited by the pipe symbol „|“) we declare the elements we want to process. We build an object list with { }, taking count and category from the top level and an array of the elements. count and category are repeated to create a proper table. In the second step we select category and the object id-s, which were on different levels previously. The tags are selected as an array. Using the syntax name: .element.element we can select elements and name them. The result of this step is a list of objects having category, id and tag in a table, suitable for writing into a relational database or processing in a data mining tool. Using jq with RapidMiner It would be useful to process these kinds of documents with jq in RapidMiner. This is what I did, using jackson-jq, a Java implementation of jq. A zip file is attached to this post, it contains the necessary libraries and two example processes.  To prepare, we need to copy the jackson-jq jar file and two dependencies into the RapidMiner Studio lib directory. Then we’re able to use the jq functionality in the built-in Groovy scripting operator (Execute Script). I created two RapidMiner processes to make the application easier. These can be used in other processes. There is one variant working on tables (example sets), here you specify the name of the input attribute containing your documents and the target attribute for the transformation result. The other variant works on Document objects, like those coming from Get Page. In both cases you specify the jq expression and set up the output options. You can indent the output, and convert the result to CSV. The CSV formatted result can be easily transformed to an example set — this is a frequent use case. If you want to see CSV output in jqplay, check „Raw Output“ and append the following: | [ .category, .id, .tag ] | @csv This creates an array (with the [] syntax) and lists the elements in the output. The result is converted with the @csv step. (The RapidMiner process does this automatically if the csv output is selected.) This, together with some practicing in jqplay, enables processes that can transform complex JSON documents to straight tables. To process different parts and structures in the document, just multiply it and apply different jq expressions on the copies.

rfuentealba

February 24

Re: Updating Anaconda and Tensorflow breaks the execute Python extension.

Hi @hughesfleming68, I'm running on 3.7.2 without problems. When you install a new Python distribution with conda update, you create a bare one as if you would have installed Miniconda. The problem isn't solved by downgrading but by installing Pandas on the new version after upgrading. What I normally do: * Install Anaconda. * Create an environment solely for RapidMiner: $ conda create --name=rapidminer python=3.7  * Activate the environment for RapidMiner: $ conda activate rapidminer * Install pandas for the new environment: $ conda install pandas * List the environments I have: $ conda env list * Copy the path of the environment I created, add /bin/python to the end and use that path to configure the Python extension. * Deactivate the environment: $ conda deactivate That way, nothing of what I do with other environments affect the one I use for RapidMiner. Hope this helps, Rodrigo.

Latest Posts

RapidMiner Community

Discussion List

• New RapidMiner research project - dealing with real-time, interactive extreme-scale analytics

  Announcement 80 views 2 comments 7 points Most recent by David_A March 5Help
• Twitter operators and their rate limits

  10 views 2 comments 0 points Most recent by sgenzer 4:07PMKnowledge Base
• Execute script Operator + include many .java files

  Responded But No Solution 16 views 1 comment 0 points Most recent by mschmitz 3:58PMHelp
• Prediction

  Responded But No Solution 15 views 1 comment 0 points Most recent by varunm1 12:52PMHelp
• AttributeTypeException, Keras changes type of label to rm_metadata("nominal", prediction")

  Responded But No Solution 31 views 1 comment 0 points Most recent by hughesfleming68 10:43AMHelp
• Applying Machine Learning to Text Mining with Amazon S3 and RapidMiner

  13 views 3 comments 1 point Most recent by sgenzer 10:10AMKnowledge Base
• "Process Documents from Data" and "Generate TF-IDF"

  Solution Accepted 13 views 2 comments 0 points Most recent by Prentice 6:50AMHelp
• Rapid Miner

  Responded But No Solution 15 views 1 comment 0 points Most recent by David_A 4:35AMHelp
• How to turn a colored histogram into a stacked bar chart?

  Solution Accepted 37 views 5 comments 0 points Most recent by sgenzer March 21Help
• How to invert the example order?

  Solution Accepted 17 views 2 comments 0 points Most recent by Newbie March 21Help
• 100,000 rows of data limit with an Educational license.

  Responded But No Solution 64 views 2 comments 1 point Most recent by p15248633 March 21Help
• Writing into multiple sheets in an Excel

  Solution Accepted 35 views 6 comments 0 points Most recent by varunm1 March 21Help
• Scoring - Using Classification models

  Responded But No Solution 44 views 6 comments 0 points Most recent by IngoRM March 21Help
• how to select important variables for my model

  Solution Accepted 33 views 3 comments 0 points Most recent by Brian_Wells March 21Help
• Process Documents from Data error

  Responded But No Solution 35 views 2 comments 0 points Most recent by Prentice March 21Help
• Using optimize parameters(Grid) on decision tree

  Solution Accepted 38 views 4 comments 0 points Most recent by Telcontar120 March 21Help
• Time Series Forecasting for many examples

  Solution Accepted 142 views 8 comments 0 points Most recent by SGolbert March 21Help
• Future product sales prediction

  Responded But No Solution 53 views 3 comments 1 point Most recent by SGolbert March 21Help
• knn medidas de similaridad y distancia

  Responded But No Solution 25 views 3 comments 0 points Most recent by SGolbert March 21Help
• Decision tree - One Branch to rule them all

  Responded But No Solution 20 views 1 comment 0 points Most recent by mschmitz March 21Help
• Industrial Data Science Conference 2019 - Dortmund, Germany

  116 views 5 comments 7 points Most recent by David_A March 21Help
• I signed up for the Education License Program but Got a Trial Version Instead - How do I fix this?

  Responded But No Solution 21 views 3 comments 0 points Most recent by Marco_Boeck March 21Help
• Filter Example with upper and lower case

  Solution Accepted 40 views 2 comments 0 points Most recent by Prentice March 21Help
• Clustering Dummy Variables

  Responded But No Solution 40 views 3 comments 0 points Most recent by Telcontar120 March 20Help
• Missing values

  Responded But No Solution 25 views 2 comments 0 points Most recent by Telcontar120 March 20Help
• With the "Read Excel" operator, how can I make it read only the visible cells?

  Solution Accepted 38 views 5 comments 0 points Most recent by Telcontar120 March 20Help
• Wrinting an excel file with multiple sheets

  Responded But No Solution 107 views 6 comments 0 points Most recent by sgenzer March 20Help
• Balancing Data based on class

  Solution Accepted 17 views 2 comments 0 points Most recent by b00122599 March 20Help
• Can't start Rapid Miner, keep receiving a error message relate with jboss.web.connector

  Responded But No Solution 16 views 1 comment 0 points Most recent by mschmitz March 20Help
• Finding correlations between outputs and inputs for large number of data

  Responded But No Solution 14 views 1 comment 0 points Most recent by varunm1 March 20Help
• I want to use a macro inside " Filter Examples" while it isn't case sensitive

  Solution Accepted 22 views 2 comments 0 points Most recent by Prentice March 20Help