spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel van der Ende <daniel.vandere...@gmail.com>
Subject Re: Spark Job not exited and shows running
Date Thu, 01 Dec 2016 07:03:04 GMT
Hi,

I've seen this a few times too. Usually it indicates that your driver
doesn't have enough resources to process the result. Sometimes increasing
driver memory is enough (yarn memory overhead can also help). Is there any
specific reason for you to run in client mode and not in cluster mode?
Having run into this a number of times (and wanting to spare the resources
of our submitting machines) we have now switched to use yarn cluster mode
by default. This seems to resolve the problem.

Hope this helps,

Daniel

On 29 Nov 2016 11:20 p.m., "Selvam Raman" <selmna@gmail.com> wrote:

> Hi,
>
> I have submitted spark job in yarn client mode. The executor and cores
> were dynamically allocated. In the job i have 20 partitions, so 5 container
> each with 4 core has been submitted. It almost processed all the records
> but it never exit the job and in the application master container i am
> seeing the below error message.
>
>  INFO yarn.YarnAllocator: Canceling requests for 0 executor containers
>  WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
>
>
>
> ​The same job i ran it for only 1000 records which successfully finished. ​
>
> Can anyone help me to sort out this issue.
>
> Spark version:2.0( AWS EMR).
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>

Mime
View raw message