spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yin Yang <>
Subject Re: TaskCompletionListener and Exceptions
Date Fri, 26 Feb 2016 23:52:38 GMT
Please see
[SPARK-13465] Add a task failure listener to TaskContext

On Sat, Dec 19, 2015 at 3:44 PM, Neelesh <> wrote:

> Hi,
>   I'm trying to build automatic Kafka watermark handling in my stream apps
> by overriding the KafkaRDDIterator, and adding a taskcompletionlistener and
> updating watermarks if task was completed (the iterator has access to
> offsets). But I found out that there is no way to listen to a task error
> inside the executors. Only the driver gets the TaskReason notification, but
> not the task completion listener. This basically means watermarks will be
> updated regardless of whether the task completed successfully or not.
> Is there any way to listen for the task failures on the executors?
> Thanks!
> -neelesh

View raw message