spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AlexG <swift...@gmail.com>
Subject spark on yarn is slower than spark-ec2 standalone, how to tune?
Date Sun, 16 Aug 2015 01:43:40 GMT
I'm using a manually installation of Spark under Yarn to run a 30 node
r3.8xlarge EC2 cluster (each node has 244Gb RAM, 600Gb SDD). All my code
runs much faster on a cluster launched w/ the spark-ec2 script, but there's
a mysterious problem with  nodes becoming inaccessible, so I switched to
using Spark under Yarn because I figure Yarn wouldn't let Spark eat up all
the resources and render a machine inaccessible. So far, this seems to be
the case. Now my code runs to completion, but much slower, so I'm wondering
how I can tune my Spark under Yarn installation to make it as fast as the
standalone spark install.

The current code I'm interested in speeding up just loads a dense 1Tb matrix
from Parquet format and then computes a low rank approximation by
essentially doing a bunch of distributed matrix multiplies. Before my code
completed in half an hour from loading to writing the output, now I expect
it to take 4 or so hours to complete.

My spark-submit options are 
  --master yarn \
  --num-executors 29 \
  --driver-memory 180G \
  --executor-memory 180G \
  --conf spark.eventLog.enabled=true \
  --conf spark.eventLog.dir=$LOGDIR \
  --conf spark.driver.maxResultSize=50G \
  --conf spark.task.maxFailures=4 \
  --conf spark.worker.timeout=1200000 \
  --conf spark.network.timeout=1200000 \
the huge timeouts were necessary on EC2 to avoid losing executors. Not sure
that they've remained necessary when switching to Yarn. 

My yarn-site.xml has the following settings:
<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>236000</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>59000</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>220000</value>
  </property>

Any suggestions?




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-on-yarn-is-slower-than-spark-ec2-standalone-how-to-tune-tp24282.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message