spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From St├ęphane Verlet <kaweahsoluti...@gmail.com>
Subject Re: Running Hive and Spark together with Dynamic Resource Allocation
Date Fri, 28 Oct 2016 21:34:06 GMT
This works for us

 <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle,spark_shuffle</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
    <value>org.apache.spark.network.yarn.YarnShuffleService</value>
  </property>





On Thu, Oct 27, 2016 at 8:13 AM, rachmaninovquartet <
rachmaninovquartet@gmail.com> wrote:

> oblem is that for Spark 1.5.2 with dynamic resource allocation to
> function properly we needed to set y
>

Mime
View raw message