RapidMiner

Highlighted
Contributor I jan_bauer
Contributor I

Performance Node Out of Memory Error

Hello,

 

when i am using the Performance Node for Classification i get an error which indicates an Out Of Memory Problem.

I optimized the Hive/Yarn/Tez settings as suggested in:

https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

and 

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/determi...

 

If i run the stream without the performance node, just the random forest model and some data preperation everything works fine.

 

The Error shown in Rapidminer is this one: (The same occurs in the hive log)

 

Exception: eu.radoop.exception.NoStackTraceOperatorException
Message: ERROR : Status: Failed Message: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1481614577214_0004_1_00, diagnostics=[Task failed, taskId=t...
Stack trace:


Cause
Exception: java.sql.SQLException
Message: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1481614577214_0004_1_00, diagnostics=[Task failed, taskId=task_1481614577214_0004_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
 at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
 at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
 at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
 at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
 at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:172)
 at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116)
 at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142)
 at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142)
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
 ... 14 more
], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
 at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
 at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
 at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
 at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
 at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:172)
 at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116)
 at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142)
 at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142)
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
 ... 14 more
], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
 at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
 at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
 at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
 at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
 at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:172)
 at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116)
 at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142)
 at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142)
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
 ... 14 more
], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:159)
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
 at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
 at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
 at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
 at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
 at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
 at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:172)
 at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116)
 at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142)
 at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142)
 at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
 ... 14 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1481614577214_0004_1_00 [Map 1] killed/failed due toSmiley SurprisedWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1481614577214_0004_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1481614577214_0004_1_01 [Reducer 2] killed/failed due toSmiley SurprisedTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
Stack trace:

  org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
  org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:392)
  eu.radoop.datahandler.hive.HiveHandler.runScriptsOrQueriesKillableAction(HiveHandler.java:1393)
  eu.radoop.datahandler.hive.HiveHandler.runScriptsOrQueriesKillable(HiveHandler.java:1204)
  eu.radoop.datahandler.hive.HiveHandler.runQueriesKillable(HiveHandler.java:1179)
  eu.radoop.datahandler.hive.HiveHandler.runQueryKillable(HiveHandler.java:1096)
  eu.radoop.RadoopTools.calculateClassificationMatrix(RadoopTools.java:897)
  eu.radoop.modeling.performance.HiveMultiClassificationPerformance.create(HiveMultiClassificationPerformance.java:63)
  eu.radoop.modeling.performance.AbstractPerformanceEvaluator.evaluate(AbstractPerformanceEvaluator.java:497)
  eu.radoop.modeling.performance.AbstractPerformanceEvaluator.evaluate(AbstractPerformanceEvaluator.java:406)
  eu.radoop.modeling.performance.AbstractPerformanceEvaluator.doWork(AbstractPerformanceEvaluator.java:390)
  com.rapidminer.operator.Operator.execute(Operator.java:1005)
  com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:77)
  com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:812)
  com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:807)
  java.security.AccessController.doPrivileged(Native Method)
  com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:807)
  eu.radoop.operator.meta.RadoopValidationChain.executeEvaluator(RadoopValidationChain.java:207)
  eu.radoop.operator.meta.RadoopValidationChain.evaluate(RadoopValidationChain.java:351)
  eu.radoop.operator.meta.SplitValidationChain.estimatePerformance(SplitValidationChain.java:66)
  eu.radoop.operator.meta.RadoopValidationChain.doWork(RadoopValidationChain.java:301)
  com.rapidminer.operator.Operator.execute(Operator.java:1005)
  com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:77)
  com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:812)
  com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:807)
  java.security.AccessController.doPrivileged(Native Method)
  com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:807)
  com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:428)
  eu.radoop.RadoopNest.doWork(RadoopNest.java:643)
  com.rapidminer.operator.Operator.execute(Operator.java:1005)
  com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:77)
  com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:812)
  com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:807)
  java.security.AccessController.doPrivileged(Native Method)
  com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:807)
  com.rapidminer.operator.meta.ParameterIteratingOperatorChain.executeSubprocess(ParameterIteratingOperatorChain.java:262)
  com.rapidminer.operator.meta.ParameterIteratingOperatorChain.getPerformanceVector(ParameterIteratingOperatorChain.java:305)
  com.rapidminer.operator.meta.ParameterIteration.evaluateParameterSet(ParameterIteration.java:230)
  com.rapidminer.operator.meta.ParameterIteration.doWork(ParameterIteration.java:189)
  com.rapidminer.operator.Operator.execute(Operator.java:1005)
  com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:77)
  com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:812)
  com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:807)
  java.security.AccessController.doPrivileged(Native Method)
  com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:807)
  com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:428)
  com.rapidminer.operator.Operator.execute(Operator.java:1005)
  com.rapidminer.Process.run(Process.java:1205)
  com.rapidminer.Process.run(Process.java:1101)
  com.rapidminer.Process.run(Process.java:1054)
  com.rapidminer.Process.run(Process.java:1049)
  com.rapidminer.Process.run(Process.java:1039)
  com.rapidminer.gui.ProcessThread.run(ProcessThread.java:65)

1 REPLY
Contributor I jan_bauer
Contributor I
Solution

Re: Performance Node Out of Memory Error

---Problem Solved----

 

i had the wrong settings!

If you follow the links above and your setting are right you should solve this error!

 

Regards,

Jan

Polls
How can RapidMiner increase participation in our new competitions?
Twitter Feed