spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alvarobrandon <>
Subject Dynamic allocation Spark
Date Fri, 26 Feb 2016 11:14:30 GMT
Hello everyone:

I'm trying the dynamic allocation in Spark with YARN. I have followed the
following configuration steps:
1. Copy the spark-*-yarn-shuffle.jar to the nodemanager classpath. "cp
/opt/spark/lib/spark-*-yarn-shuffle.jar /opt/hadoop/share/hadoop/yarn"
2. Added the shuffle service of spark in yarn-site.xml
    <description>shuffle implementation</description>
3. Enabled the class for the shuffle service in yarn-site.xml
    <description>enable the class for dynamic allocation</description>
4. Activated the dynamic allocation in the spark defaults
spark.dynamicAllocation.enabled 	true
spark.shuffle.service.enabled 	true

When I launch my application it just stays in the queue accepted but it
never actually runs.
16/02/26 11:11:46 INFO yarn.Client: Application report for
application_1456482268159_0001 (state: ACCEPTED)

Am I missing something?

Thanks in advance as always 

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message