spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hao Ren <inv...@gmail.com>
Subject Re: Can not kill driver properly
Date Mon, 21 Mar 2016 16:25:50 GMT
Update:

I am using --supervise flag for fault tolerance.



On Mon, Mar 21, 2016 at 4:16 PM, Hao Ren <invkrh@gmail.com> wrote:

> Using spark 1.6.1
> Spark Streaming Jobs are submitted via spark-submit (cluster mode)
>
> I tried to kill drivers via webUI, it does not work. These drivers are
> still running.
> I also tried:
> 1. spark-submit --master <master-url> --kill <driver-id>
> 2. ./bin/spark-class org.apache.spark.deploy.Client kill <master url>
> <driver ID>
>
> Neither works. The workaround is to ssh to the driver node, then kill -9
> ...
> jsp shows the same classname DriverWrapper, so need to pick carefully...
>
> Any idea why this happens ?
> BTW, my streaming job's batch duration is one hour. So do we need to wait
> for job processing to kill kill driver ?
>
> --
> Hao Ren
>
> Data Engineer @ leboncoin
>
> Paris, France
>



-- 
Hao Ren

Data Engineer @ leboncoin

Paris, France

Mime
View raw message