Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

Process Pushdown error

asmahassani08asmahassani08 Member Posts: 31 Contributor II
edited December 2018 in Help
I'm working on radoop. I'm using the sparkRM to extract association rules. for that, I apply the fpgrowth in the sparkRM operator subprocess.
I have these error : SparkRM: The driver resource request has not yet been granted to the Spark job.
joined the process and the error.

Tagged:

Answers

  • asmahassani08asmahassani08 Member Posts: 31 Contributor II
    I changed the pushdown process to sparkRM operator and have problems: 

    what does it mean: The driver resource request has not yet been granted to the Spark job.
    In rapidminer's logs, I have this  SparkRM (2): Yarn application state of application_1544000400659_0013: ACCEPTED 
    in the yarn :
    ContainersMonitorImpl: Memory usage of ProcessTree container_1544000400659_0013_01_000001 for container-id container_1544000400659_0013_01_000001: 278.2 MB of 2 GB physical memory used; 455.0 MB of 4.2 GB virtual memory used
    the job is accepted and not yet running, what can be the source of the problem?

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    cc @asimon
  • toth_zsolt_bmetoth_zsolt_bme Member Posts: 1 Learner III
    Hi,

    this can mean multiple things, my guess is that your cluster doesn't have enough free resources to start the job. You can try to play a bit with the Spark Resource Allocation Policy settings in the Radoop Connection, and the cluster resources, driver memory, executor memory settings in the operator params, e.g. try to decrease the cluster resources%.
    If you have access to the yarn CLI, you can do "yarn logs -applicationId application_1544000400659_0013" to see the full log of the application.


  • asmahassani08asmahassani08 Member Posts: 31 Contributor II
    edited December 2018
    thank you for your response, the problem is that I'm new with Hadoop and spark and I don't know what and how should I configure things.
    follow yarn-site.xml.
    this is spark-default.conf : 
    <div class="Quote"># spark.master&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;yarn<br># spark.eventLog.enabled&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;true<br># spark.eventLog.dir&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;hdfs://namenode:8021/directory<br># spark.serializer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;org.apache.spark.serializer.KryoSerializer<br># spark.driver.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4G<br># spark.driver.cores					1<br># spark.yarn.driver.memoryOverhead	384MB<br># spark.yarn.am.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1G<br># spark.yarn.am.cores				1<br># spark.yarn.am.memoryOverhead		384MB<br># spark.yarn.nodemanager.resource.cpu-vcores&nbsp; &nbsp;1<br># spark.yarn.nodemanager.resource.memory-mb 2048MB<br># spark.executor.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2G<br># spark.executor.instances			1<br># spark.executor.cores				1<br># spark.yarn.executor.memoryOverhead	384MB<br># spark.executor.extraJavaOptions&nbsp; -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"</div>
    Spark Resource Allocation policy = static, default configuration 
    driver memory = 1024MB
    in the diagnostic of the application i have as follow: 
    hope that I have a solution and get the things working. 
Sign In or Register to comment.