spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luis Ángel Vicente Sánchez <langel.gro...@gmail.com>
Subject Re: Kafka Receiver not recreated after executor died
Date Tue, 16 Dec 2014 16:56:42 GMT
It seems to be slightly related to this:

https://issues.apache.org/jira/browse/SPARK-1340

But in this case, it's not the Task that is failing but the entire executor
where the Kafka Receiver resides.

2014-12-16 16:53 GMT+00:00 Luis Ángel Vicente Sánchez <
langel.groups@gmail.com>:
>
> Dear spark community,
>
>
> We were testing a spark failure scenario where the executor that is
> running a Kafka Receiver dies.
>
> We are running our streaming jobs on top of mesos and we killed the mesos
> slave that was running the executor ; a new executor was created on another
> mesos-slave but according to the Driver UI, the Kafka receiver location is
> still on the dead slave.
>
> It seems that if the executor where the receiver was running dies, spark
> on top of mesos is not able to create a new receiver on a different
> executor to continue working.
>
> Is that a known issue? Is there a way we could ensure that the receiver
> would be recreated?
>
> Kind regards,
>
> Luis Vicente
>

Mime
View raw message