spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yong Zhang <>
Subject Re: Tuning level of Parallelism: Increase or decrease?
Date Wed, 03 Aug 2016 11:01:39 GMT
Data Locality is part of job/task scheduling responsibility. So both links you specified originally
are correct, one is for the standalone mode comes with Spark, another is for the YARN. Both
have this ability.

But YARN, as a very popular scheduling component, comes with MUCH, MUCH more features than
the Standalone mode. You can research more on google about it.


From: Jestin Ma <>
Sent: Tuesday, August 2, 2016 7:11 PM
To: Jacek Laskowski
Cc: Nikolay Zhebet; Andrew Ehrlich; user
Subject: Re: Tuning level of Parallelism: Increase or decrease?

Hi Jacek,
I found this page of your book here:
Data Locality ยท Mastering Apache Spark<>
Spark relies on data locality, aka data placement or proximity to data source, that makes
Spark jobs sensitive to where the data is located. It is therefore important ...

which says:  "It is therefore important to have Spark running on Hadoop YARN cluster<>
if the data comes from HDFS. In Spark on YARN<>
Spark tries to place tasks alongside HDFS blocks."

So my reasoning was that since Spark takes care of data locality when workers load data from
HDFS, I can't see why running on YARN is more important.

Hope this makes my question clearer.

On Tue, Aug 2, 2016 at 3:54 PM, Jacek Laskowski <<>>
On Mon, Aug 1, 2016 at 5:56 PM, Jestin Ma <<>>
> Hi Nikolay, I'm looking at data locality improvements for Spark, and I have
> conflicting sources on using YARN for Spark.
> Reynold said that Spark workers automatically take care of data locality
> here:
> However, I've read elsewhere
> (
> that Spark on YARN increases data locality because YARN tries to place tasks
> next to HDFS blocks.
> Can anyone verify/support one side or the other?

Hi Jestin,

I'm the author of the latter. I can't seem to find how Reynold
"conflicts" with what I wrote in the notes? Could you elaborate?

I certainly may be wrong.

Jacek Laskowski
Mastering Apache Spark 2.0
Follow me at

View raw message