spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <>
Subject Re: spark application running in yarn client mode is slower than in local mode.
Date Mon, 09 Apr 2018 06:12:00 GMT
Probably network / shuffling cost? Or broadcast variables? Can you provide more details what
you do and some timings?

> On 9. Apr 2018, at 07:07, Junfeng Chen <> wrote:
> I have wrote an spark streaming application reading kafka data and convert the json data
to parquet and save to hdfs. 
> What make me puzzled is, the processing time of app in yarn mode cost 20% to 50% more
time than in local mode. My cluster have three nodes with three node managers, and all three
hosts have same hardware, 40cores and 256GB memory. .
> Why? How to solve it? 
> Regard,
> Junfeng Chen

To unsubscribe e-mail:

View raw message