spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: SparkContext.stop() ?
Date Fri, 31 Oct 2014 18:03:48 GMT
You don't have to call it if you just exit your application, but it's useful for example in
unit tests if you want to create and shut down a separate SparkContext for each test.

Matei

> On Oct 31, 2014, at 10:39 AM, Evan R. Sparks <evan.sparks@gmail.com> wrote:
> 
> In cluster settings if you don't explicitly call sc.stop() your application may hang.
Like closing files, network connections, etc, when you're done with them, it's a good idea
to call sc.stop(), which lets the spark master know that your application is finished consuming
resources.
> 
> On Fri, Oct 31, 2014 at 10:13 AM, Daniel Siegmann <daniel.siegmann@velos.io <mailto:daniel.siegmann@velos.io>>
wrote:
> It is used to shut down the context when you're done with it, but if you're using a context
for the lifetime of your application I don't think it matters.
> 
> I use this in my unit tests, because they start up local contexts and you can't have
multiple local contexts open so each test must stop its context when it's done.
> 
> On Fri, Oct 31, 2014 at 11:12 AM, ll <duy.huynh.uiv@gmail.com <mailto:duy.huynh.uiv@gmail.com>>
wrote:
> what is it for?  when do we call it?
> 
> thanks!
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-stop-tp17826.html
<http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-stop-tp17826.html>
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <mailto:user-unsubscribe@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org <mailto:user-help@spark.apache.org>
> 
> 
> 
> 
> -- 
> Daniel Siegmann, Software Developer
> Velos
> Accelerating Machine Learning
> 
> 440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
> E: daniel.siegmann@velos.io <mailto:daniel.siegmann@velos.io> W: www.velos.io <http://www.velos.io/>


Mime
View raw message