spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: How to track batch jobs in spark ?
Date Thu, 06 Dec 2018 10:40:09 GMT
Thanks for all responses.

1) I am not using YARN. I am using Spark Standalone.
2) yes I want to be able to kill the whole Application.
3) I want to be able to monitor the status of the Application which is
running a batch query and expected to run for an hour or so, therefore, I
am looking for some mechanism where I can monitor the progress like a
percentage or something.

Thanks!


On Wed, Dec 5, 2018 at 3:12 PM Mark Hamstra <mark@clearstorydata.com> wrote:

> That will kill an entire Spark application, not a batch Job.
>
> On Wed, Dec 5, 2018 at 3:07 PM Priya Matpadi <pmatpadi@gmail.com> wrote:
>
>> if you are deploying your spark application on YARN cluster,
>> 1. ssh into master node
>> 2. List the currently running application and retreive the application_id
>>     yarn application --list
>> 3. Kill the application using application_id of the form
>> application_xxxxx_xxxx from output of list command
>>         yarn application --kill <application_id>
>>
>> On Wed, Dec 5, 2018 at 1:42 PM kant kodali <kanth909@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> How to track batch jobs in spark? For example, is there some id or token
>>> i can get after I spawn a batch job and use it to track the progress or to
>>> kill the batch job itself?
>>>
>>> For Streaming, we have StreamingQuery.id()
>>>
>>> Thanks!
>>>
>>

Mime
View raw message