spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rachmaninovquartet <>
Subject Running Hive and Spark together with Dynamic Resource Allocation
Date Thu, 27 Oct 2016 14:13:21 GMT

My team has a cluster running HDP, with Hive and Spark. We setup spark to
use dynamic resource allocation, for benefits such as not having to hard
code the number of executors and to free resources after using. Everything
is running on YARN.

The problem is that for Spark 1.5.2 with dynamic resource allocation to
function properly we needed to set yarn.nodemanager.aux-services in
yarn-site.xml to spark_shuffle, but this breaks hive (1.2.1), since it is
looking for auxService:mapreduce_shuffle. Does any one know of a way to
configure in order to have both services running smoothly?



View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe e-mail:

View raw message