spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: spark-submit on YARN is slow
Date Mon, 08 Dec 2014 01:39:04 GMT
Hi,

thanks for your responses!

On Sat, Dec 6, 2014 at 4:22 AM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
>
> What version are you using?  In some recent versions, we had a couple of
> large hardcoded sleeps on the Spark side.
>

I am using Spark 1.1.1.

As Andrew mentioned, I guess most of the 10 seconds waiting time probably
comes from YARN itself. (Other YARN applications also take a while to start
up.) I'm just really puzzled about what exactly takes so long there... for
a job that runs an hour or so, that is of course negligible, but I am
starting up an instance per client to do interactive job processing *for
this client*, and it feels like "yeah, thanks for logging in, now please
wait a while until you can actually use the program", that's a bit
suboptimal.

Tobias

Mime
View raw message