spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <>
Subject Re: executors not created yarn-cluster mode
Date Wed, 08 Oct 2014 20:38:20 GMT
Hi Jamborta,

It could be that your executors are requesting too much memory. I'm not
sure why it works in client mode but not in cluster mode, however. Have you
checked the RM logs for messages that complain about container memory
requested being too high? How much memory is each of your container asking
for (check AM logs for this)? As a sanity check you could try lowering the
executor memory to see if that's in fact the issue. Another factor at play
here is "spark.yarn.executor.memoryOverhead", which adds to the amount of
memory requested and is 348MB by default.

Let me know if you find anything,

2014-10-08 12:00 GMT-07:00 jamborta <>:

> Hi all,
> I have a setup that works fine in yarn-client mode, but when I change that
> to yarn-cluster, the executors don't get created, apart from the driver (it
> seems that it does not even appear in yarn's resource manager). There is
> nothing in the spark logs, either (even when debug is enabled). When I try
> to submit something to the sparkcontext I get the following error:
> Initial job has not accepted any resources; check your cluster UI to ensure
> that workers are registered and have sufficient memory
> Just wondering where can I find the logs that show the executor creation
> process? I looked at yarn logs, also the spark event logs, couldn't find
> anything.
> many thanks,
> --
> View this message in context:
> Sent from the Apache Spark User List mailing list archive at
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message