spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <t...@databricks.com>
Subject Re: spark core/streaming doubts
Date Wed, 08 Jul 2015 18:13:16 GMT
Responses inline.

On Wed, Jul 8, 2015 at 10:26 AM, Shushant Arora <shushantarora09@gmail.com>
wrote:

> 1.Does creation of read only singleton object in each map function is same
> as broadcast object as singleton never gets garbage collected unless
> executor gets shutdown ? Aim is to avoid creation of complex object at each
> batch interval of a spark streaming app.
>
> No, objects created in a map function are transient objects in the
executor, which gets GCed as long as you dont set up permanent references
to those objects (through singletons and statics) that prevent GC.


>
> 2.why JavaStreamingContext 's sc () method is deprecated? Whats the other
> way to access spark context to broadcast a variable then?
>     jssc.sc().broadcast(filter);.
>
> jssc.sparkContext()


> 3.Does in streamapp processing executors (executors other than
> receivers)stay 24*7 till streaming app is alive?
> And task are allocated in threads on these executors?
>
> Executors stay up as long as the SparkContext is not stopped. This is true
for any Spark application, not just Spark Streaming applications.
But executors can fail and can get restarted. So its not correct to rely on
24/7 availability. So you have to plan for faults and recovery if you have
to do some custom stateful stuff.



> Thanks
> Shushant
>
>
>

Mime
View raw message