spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thakrar, Jayesh" <>
Subject Re: How to track batch jobs in spark ?
Date Thu, 06 Dec 2018 15:40:27 GMT
See if helps.

Essentially whether you are running an app as spark-shell, via spark-submit (local, Spark-Cluster,
YARN, Kubernetes, mesos), the driver will provide a UI on port 4040.

You can monitor via the UI and via a REST API

E.g. running a job locally on your laptop, you can run something like this

To see the “jobs”, you can use something like this
(local-1544110095543 is just the id of my spark-shell on my laptop which I got from the command

If you are only look for job completion, you can just monitor if there is a listener on the
Once job completes/fails, the driver and the listener will exit and hence it means the job
is complete.

As far as giving you the percentage complete for the application, there is no such thing as
unlike mapreduce, a Spark app is not a single step/job.
Using the REST API, you can see which job/stage is running and determine what percentage of
your job is complete.
Even when getting the stage info, you only get the number of “tasks” complete v/s percentage

From: kant kodali <>
Date: Thursday, December 6, 2018 at 4:40 AM
To: Mark Hamstra <>
Cc: <>, "user @spark" <>
Subject: Re: How to track batch jobs in spark ?

Thanks for all responses.

1) I am not using YARN. I am using Spark Standalone.
2) yes I want to be able to kill the whole Application.
3) I want to be able to monitor the status of the Application which is running a batch query
and expected to run for an hour or so, therefore, I am looking for some mechanism where I
can monitor the progress like a percentage or something.


On Wed, Dec 5, 2018 at 3:12 PM Mark Hamstra <<>>
That will kill an entire Spark application, not a batch Job.

On Wed, Dec 5, 2018 at 3:07 PM Priya Matpadi <<>>
if you are deploying your spark application on YARN cluster,
1. ssh into master node
2. List the currently running application and retreive the application_id
    yarn application --list
3. Kill the application using application_id of the form application_xxxxx_xxxx from output
of list command
        yarn application --kill <application_id>

On Wed, Dec 5, 2018 at 1:42 PM kant kodali <<>>
Hi All,

How to track batch jobs in spark? For example, is there some id or token i can get after I
spawn a batch job and use it to track the progress or to kill the batch job itself?

For Streaming, we have

View raw message