spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imre Nagi <imre.nagi2...@gmail.com>
Subject Stop spark application when the job is complete.
Date Thu, 17 Mar 2016 04:17:59 GMT
Hi,

I have a spark application for batch processing in standalone cluster. The
job is to query the database and then do some transformation, aggregation,
and several actions such as indexing the result into the elasticsearch.

If I dont call the sc.stop(), the spark application wont stop and take will
keep the resource used by the application. In the other hand, if I call the
sc.stop(), the spark app will be stopped before it query the database and
do further processing.

Can anyone help me to give best practice in stopping the spark application
when the job is complete?

Thanks,
Imre

Mime
View raw message