Options

Process Pushdown error

asmahassani08asmahassani08 Member Posts: 31 Contributor II
edited December 2018 in Help
I'm working on radoop. I'm using the sparkRM to extract association rules. for that, I apply the fpgrowth in the sparkRM operator subprocess.
I have these error : SparkRM: The driver resource request has not yet been granted to the Spark job.
joined the process and the error.

Tagged:

Answers

  • Options
    asmahassani08asmahassani08 Member Posts: 31 Contributor II
    I changed the pushdown process to sparkRM operator and have problems: 

    what does it mean: The driver resource request has not yet been granted to the Spark job.
    In rapidminer's logs, I have this  SparkRM (2): Yarn application state of application_1544000400659_0013: ACCEPTED 
    in the yarn :
    ContainersMonitorImpl: Memory usage of ProcessTree container_1544000400659_0013_01_000001 for container-id container_1544000400659_0013_01_000001: 278.2 MB of 2 GB physical memory used; 455.0 MB of 4.2 GB virtual memory used
    the job is accepted and not yet running, what can be the source of the problem?

  • Options
    sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    cc @asimon
  • Options
    toth_zsolt_bmetoth_zsolt_bme Member Posts: 1 Contributor I
    Hi,

    this can mean multiple things, my guess is that your cluster doesn't have enough free resources to start the job. You can try to play a bit with the Spark Resource Allocation Policy settings in the Radoop Connection, and the cluster resources, driver memory, executor memory settings in the operator params, e.g. try to decrease the cluster resources%.
    If you have access to the yarn CLI, you can do "yarn logs -applicationId application_1544000400659_0013" to see the full log of the application.


  • Options
    asmahassani08asmahassani08 Member Posts: 31 Contributor II
    edited December 2018
    thank you for your response, the problem is that I'm new with Hadoop and spark and I don't know what and how should I configure things.
    follow yarn-site.xml.
    this is spark-default.conf : 
    <div class="Quote"># spark.master&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;yarn<br># spark.eventLog.enabled&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;true<br># spark.eventLog.dir&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;hdfs://namenode:8021/directory<br># spark.serializer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;org.apache.spark.serializer.KryoSerializer<br># spark.driver.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4G<br># spark.driver.cores					1<br># spark.yarn.driver.memoryOverhead	384MB<br># spark.yarn.am.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1G<br># spark.yarn.am.cores				1<br># spark.yarn.am.memoryOverhead		384MB<br># spark.yarn.nodemanager.resource.cpu-vcores&nbsp; &nbsp;1<br># spark.yarn.nodemanager.resource.memory-mb 2048MB<br># spark.executor.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2G<br># spark.executor.instances			1<br># spark.executor.cores				1<br># spark.yarn.executor.memoryOverhead	384MB<br># spark.executor.extraJavaOptions&nbsp; -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"</div>
    Spark Resource Allocation policy = static, default configuration 
    driver memory = 1024MB
    in the diagnostic of the application i have as follow: 
    hope that I have a solution and get the things working. 
Sign In or Register to comment.