Highest Rated Posts

varunm1

July 9

Re: Is there anyway of customizing Rapidminer to show your name when taking a screen shot?

Hello @BROOT You can change your computer name to your name. This will appear on the rapidminer window on top. 

IngoRM

July 4

Re: Feature Selection within CV: Which features are finally selected?

Ok, let's try differently then (and sorry when I got way back to the basics on this one...) Let's assume that you have a data set with 1,000 rows.  You want to build a ML on this data and put it into production.  Naturally, you build the model on as much data as possible (all 1,000 rows) since then the model can learn from most information which in general is a good idea. But not so fast: how can you be sure how well the model will perform for NEW data in the future?  That's why we validate on test data (at least with a split validation, but if circumstances allow you probably go with a cross-validation.  Great, 90% accurate, let's put the model into production. Please note, that I did NOT say anything optimizing the model here, the only purpose of the validation is to estimate the future performance of the ORIGINAL model built on all 1,000 rows. For this reason, RapidMiner offers the "mod" (short for Model) port at the cross validation so that you can make sure that the validated model and the model built on the complete data is using the same hyperparameters.  This is just a convenience feature since this is such a frequent pattern but again has nothing to do with any attempt of improving the model as part of the validation. So now let's add feature selection to this discussion.  The argument is the very same as above, but this time we replace "Model" with "Feature Selection and Model" in the flow: (Quote) You see?  The feature selection becomes a part of the model building.  It is validated in exactly the same way as a model without FS. Exactly for the reason you have mentioned since you would have information leakage if you do it before. But after validation, and for the same reasons as for a model without FS, you also build the model AND the FS on the complete data for the production model in the end.  Again, the whole point of the validation is to estimate how well it works, not to find good feature sets, parameters or anything else. Hope this helps, Ingo

IngoRM

June 22

Re: Cross Validation or Split Validation

Hi @Arupriya_Sen What sounds like an easy question does not have an easy answer.  Many will say "go with cross validation by all means", but I see this a bit more differentiated.  Below are some thoughts, but if you want to read a more in-depth discussion I would suggest to check out this white paper here: https://rapidminer.com/resource/correct-model-validation/ There are really three important things when it comes to validation (the third one covers your specific question): * Never ever calculate training errors.  They are only misleading and not useful for model valuation at all.  Check out a white paper to see some easy examples for this BTW.  I assume you know this, but I point it out again mainly for point (2) below. * Treat preprocessing as part of the model building.  Many data preprocessing steps (e.g. normalization, feature engineering, encoding and many more) will take into account the complete data set for their calculations and target model improvement.  Because of that, those operations actually become part of the model building.  The same is true for parameter optimization and all other model-optimizing techniques.  Following rule (1) above, you have to do a proper validation of those techniques on a test set as well, otherwise your model will look better in validation than it actually will perform in production.  This topic is more complex and I highly recommend to check out the white paper for more information. * Pick the validation scheme most appropriate for your situation.  We have established that you need an independent test and that you need to validate (most of) your preprocessing.  This leads us to the problem of picking a good validation scheme out of the many existing ones: split validation, multiple hold-out sets, cross validation, leave-one-out, bootstrapping etc.  In practice, I recommend the following three techniques depending on your situation: * Split validation: use this if your data is very large, training times are very long, you do not have a lot of time, you perform very complex preprocessing processes with multiple nested steps, the use case you are working on is not that critical so that you can accept some uncertainty about the robustness of your models. * Cross validation: use this if you want to get the most thoroughly tested models, your data is small, your processes are not very complex so that you can easily embed them in one or multiple nested cross validations, total runtime is not an issue for you, the use case is life-or-death important. * Split validation with a robust multiple hold-out set validation: good compromise between both approaches which delivers estimation qualities similar to those of cross validations without the drawback of their long runtimes, also works with more complex process designs.  The quality of the validations is good enough for most business applications. One more word on technique (3) above: the idea is that you split off a validation set of some decent size (typically around 40% of the data) and then perform all preprocessing (where necessary) and model building on the training part (the other 60% or so) as usual.  But for the actual validation, you then perform a multiple non-overlapping hold-out set validation on those 40% you left out before.  In addition, you skip the outliers in the outcome of the hold-out runs.  That means that you build, let's say, 7 hold-out sets out of the 40% validation set, calculate the error rates on those 7 sets, skip the ones with highest and lowest error rates, and finally compute average and standard deviations on the rest.  This means that you will get some of the main advantages of cross validation like reducing the dependency on a specific hold-out sample, getting a standard deviation as an indicator for model robustness etc.  But since scoring is typically fast and training is slow, you also get the biggest advantage of split validation, which is shorter runtimes.  As I mentioned before, this technique also deals better with complex process designs. This third technique is much less known than the other two, but it delivers about the same quality as cross validations with significant lower runtimes.  In our tests, the differences in the estimations with this technique and cross validation was statistically not significant in 19 out of 20 data sets.  For all those reasons, we are also using this technique in Auto Model for the model validations. Final comment: I hope the guidance about when to use which validation technique is helpful, but personally I even think that this question is less important than being thorough with validating EVERYTHING correctly (point (2) above), including the preprocessing.  If you do this wrong, the differences of error rates you see in the validated results and what you actually get in production can easily be 10x higher than the differences caused by the different validation schemes. Hope this helps, Ingo

rfuentealba

June 19

Re: General Results

Hello @naxiota, Ok, let's see how to help you. Since you are learning, I prefer you to see and repeat with me: You have one file. Here I used a version of the Holy Bible (please see my disclaimer at the end), so I used the Open File operator to read it, extract the content with the Read Document operator and then pass it as a document to Process Documents. Inside the Process Documents super-operator (the kinds of operators that let you put stuff inside those), I did this: Well... * Tokenizing is boring because it's more an internal representation of words, you won't get meaningful things from that. * Also, it is a good idea to Transform Cases because it brings together words with mixed cases (those are not considered the same under normal comparison algorithms). * Now on to something useful: Let's extract all the Nouns from the content. You may be able to do many other things, but this is a simple exercise: you can extract the POS tags using this https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html Now you can do whatever you want with natural language processing. I added the file but please try figuring things out with the pictures I sent and the explanations first, it helps better. Hope this helps, Rodrigo.

Marco_Boeck Team Lead Software Engineering

June 14

Re: Sankey plot

Hi, Nope, but I used this simple set to test it Regards, Marco

josephreddy07

June 13

RapidMiner & SAP HANA DB Integration using ODBC

Objective:  If you have SAP HANA data base which stores all the enterprise transactional data and want to apply predictive/machine learning algorithms on the HANA data base tables or views using Rapid Miner. This blog gives you the steps to connect SAP HANA data base from Rapid Miner and retrieve tables/views/procedures data and apply Rapid Miner statistical algorithms or machine learning techniques to get the insights of data.  Back Ground and use case:Rapid Miner is a data science platform for teams that unites data prep, machine learning, text mining and predictive model deployment. It is used for business and commercial applications as well as for research, education, training, rapid prototyping, and application development and supports all steps of the machine learning process including data preparation, results visualization, model validation and optimization. SAP HANA is an in-memory, column-oriented, relational database management system developed and marketed by SAP SE. Its primary function as a database server is to store and retrieve data as requested by the applications.With the SAP HANA 2.0 Data Platform of structured and unstructured data,I wanted to do the proof of concept on Text analysis, Market basket analysis, Customer segmentation and other data science use cases using Rapid Miner Studio. The below step by step procedure helps to  connect HANA DB tables/views/procedures from Rapid miner studio. SAP HANA DB & Rapid Miner Integration steps:  1.     Prerequisites : You must have below before trying your hands on Rapid miner models on HANA data.      HANA CLIENT Tools (Which installs required drivers in the client machine)      Rapid Miner Studio Client (Interface to develop Rapid miner models)       User Account in HANA DB with read access on DB Objects.  Optional: Install HANASTUDIO/WEB IDE to analyze or preview SAP HANA data.  2.     Once you Install HANA Client tools, you will see ngdbc.jar file in the below path              C:\Program Files\SAP\hdbclient              Place this ngdbc.jar file below Rapid miner installation folder path              C:\Program Files\RapidMiner\RapidMiner Studio\lib\jdbc  3.     Open the Rapid Miner studio and create data base driver . Go to manage data base drivers and enter the following parameters as shown in picture            Name of driver: HANA_JDBC             Jar file: C:\Program Files\RapidMiner\RapidMiner Studio\lib\jdbc\ngdbc.jar              Port : 30115              Driver class: com.sap.db.jdbc.Driver   4..     Now go to manage data base connections , create data base connection with below parameters. Select previously created driver in the data base system Give the HANA system host, port ,user id, password.  Do the test connection- you should see success message.   You can use tables or views or write SQL script  to retrieve the data. 5.Now use db connect process to import the data and process the data using rapid miner models. From read data base parameters, you can connect tables or views or you can write custom HANA SQL Query.   Writing custom SQL query : Now use other operators in Rapid miner and feed this data based on your usecase.

Latest Posts

RapidMiner Community

Discussion List

• Hey :) We're looking for user feedback

  Announcement 180 views 10 comments 7 points Most recent by EgiShiaku July 12Help
• See the python script of Keras model

  Responded But No Solution 58 views 5 comments 0 points Most recent by varunm1 8:11PMHelp
• Combined SMOTE Operator

  Solution Accepted 58 views 5 comments 0 points Most recent by jacobcybulski 7:24PMHelp
• How to Use Macros

  457 views 9 comments 7 points Most recent by sgenzer 4:23PMKnowledge Base
• NaN Values on Logistic Regression Output

  Responded But No Solution 22 views 2 comments 0 points Most recent by sgenzer 3:41PMHelp
• Decision tree using Auto model

  Solution Accepted 110 views 18 comments 0 points Most recent by sgenzer 3:32PMHelp
• Date importing in different time zones

  Responded But No Solution 42 views 6 comments 0 points Most recent by kayman July 20Help
• How can I download models depicted in Rapidminer documentation

  Responded But No Solution 22 views 3 comments 0 points Most recent by varunm1 July 20Help
• New deep learning extension, how to increase the model's precision

  Responded But No Solution 48 views 8 comments 0 points Most recent by jacobcybulski July 19Help
• 9.4 beta Log operator issue

  Responded But No Solution 20 views 2 comments 0 points Most recent by Noel July 19Help
• Stop word removal with operator toolbox extract topics from data(LDA)

  Solution Accepted 18 views 3 comments 0 points Most recent by mschmitz July 19Help
• Multi Class Labels

  Responded But No Solution 305 views 15 comments 0 points Most recent by varunm1 July 19Help
• how to split date

  Responded But No Solution 27 views 3 comments 0 points Most recent by halaalrobassy July 19Help
• Error explanation

  Responded But No Solution 22 views 6 comments 0 points Most recent by varunm1 July 19Help
• Problem with reading files from Azure Blob Storage

  New Question 10 views 0 comments 0 points Started by Papad July 19 Help
• how to get length of numbers in column

  Solution Accepted 23 views 5 comments 0 points Most recent by halaalrobassy July 19Help
• X-Means output

  Responded But No Solution 25 views 3 comments 0 points Most recent by IngoRM July 19Help
• RapidMiner Studio 9.4 Beta Java issue

  Solution Accepted 27 views 2 comments 0 points Most recent by Noel July 18Help
• Possibility to save plot with animation.

  Solution Accepted 21 views 4 comments 0 points Most recent by Marco_Boeck July 18Help
• RapidMiner using TeamViewer

  Solution Accepted 51 views 7 comments 0 points Most recent by varunm1 July 18Help
• Hybrid

  Solution Accepted 42 views 3 comments 0 points Most recent by Telcontar120 July 18Help
• Maintaining Data Resolution with in-equal sampling frequencies

  Responded But No Solution 22 views 1 comment 0 points Most recent by Telcontar120 July 18Help
• Download from Azure Blob Storage

  Solution Accepted 15 views 6 comments 0 points Most recent by Papad July 18Help
• Group examples together in a loop

  Solution Accepted 40 views 4 comments 0 points Most recent by huayu July 17Help
• No puedo ver todos los resultados ?

  Responded But No Solution 17 views 1 comment 0 points Most recent by rfuentealba July 17Help
• Process failed exception, any clue?

  Solution Accepted 63 views 8 comments 0 points Most recent by confusedMonMon July 17Help
• Limitations with the RapidMiner Educational License Program

  Solution Accepted 30 views 2 comments 0 points Most recent by User142977 July 17Help
• Keras Sample bug??

  Solution Accepted 17 views 1 comment 0 points Most recent by varunm1 July 16Help
• Averaging attribute weights

  Solution Accepted 22 views 2 comments 0 points Most recent by varunm1 July 16Help
• Connecting to Teradata

  Responded But No Solution 24 views 5 comments 0 points Most recent by Marco_Boeck July 16Help
• How to Connect a Teradata Database to Studio?

  201 views 3 comments 2 points Most recent by Marco_Boeck July 16Knowledge Base