Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.
Parallel processing inside of a loop operator?
I have never seen this before, but there seems to be parallel processing inside of a loop examples operator. I know that in some operators one is able to select parallel execution, but I was always of the opinion it was not possible in Loop Example?
Tagged:
0
Best Answer
-
David_A Administrator, Moderator, Employee, RMResearcher, Member Posts: 297 RM ResearchHi,the Loop Examples operator itself does not execute in parallel. But of course if you run any parallelized Operator inside the loop, it can be executed in parallel.Is there a specific reason for your question?Best,
David6
Answers
Thanks.
Scott
It's pretty sneaky - the 'exa' port will RESEND the data back to the input 'exa' port of Loop Examples for each iteration; the 'out' port will not. So after your first iteration the way you had it, the data coming into Extract Macro (7) was the data that went out of Join (2) after the previous iteration.
Clear as mud? That's not a bug - that's just the way Loop Examples works.
Scott
[EDIT FWIW the help panel does try to explain this...]
One important thing to note about this operator is the behavior of the example setoutput port of its subprocess. The subprocess is given the ExampleSet provided at the outer example setinput port in the first iteration. If the example setoutput port of the subprocess is connected the ExampleSet delivered here in the last iteration will be used as input for the following iteration. If it is not connected the original ExampleSet will be delivered in all iterations.anw
Cause, I did not pick up anywhere that this is how the operator works. So yip, pretty muddy.
Scott
One important note on the behaviour of the example set output port for Loop Examples, the first iteration of Loop Examples uses the ExampleSet provided at the outer example set input port, for the next iteration if the output from the process is connected to the example set output port and not to the output port then the ExampleSet delivered to the example set port will be used for this iteration. Connecting the output to the output port means the process will then use the input port ExampleSet in the next iteration. If the output is not connected to either of the ports then the input port ExampleSet will be delivered in all iterations.
https://docs.rapidminer.com/latest/studio/operators/utility/process_control/loops/loop_examples.html
Can you provide the list of parallelized Operator?
Can we run spark in parallel mode in standard Loop Values?
I have tried using standard “Loop Values” with enable parallel execution by Inside the loop values operator, using Radoop nest with SparkRM as shown below
I ran this workflow on AI hub server, but I got error. If I use the same flow without enable parallel execution on Loop values operator. The flow works smoothly without error but it is quite slow.
Any suggestion?
Dortmund, Germany