spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Pascal Billaud ...@tellapart.com>
Subject Re: Spark streaming and executor object reusage
Date Sat, 07 Mar 2015 16:36:07 GMT
Thanks a lot.

Sent from my iPad

> On Mar 7, 2015, at 8:26 AM, Sean Owen <sowen@cloudera.com> wrote:
> 
>> On Sat, Mar 7, 2015 at 4:17 PM, Jean-Pascal Billaud <jp@tellapart.com> wrote:
>> So given this let's go a bit further. Imagine my static factory provides a stats
collector that my various map() code would use to export some metrics while mapping tuples.
This stats collector comes with a timer that flush the stats buffer once every 5mn for instance.
Given that executor JVM should not be reinitialized over the life of the DStream in practice,
it is totally reasonable to assume that this timer will be able to do his job. Right?
> 
> Ignoring failures, the executors should not be restarted over the life
> of the streaming app, yes. You will have, potentially, many executors
> at once of course.
> 
>> Given what you said earlier that totally makes sense. In general is there any spark
architecture documentation other than the code that gives a good overview of the thing we
talked about?
> 
> http://spark.apache.org/docs/latest/cluster-overview.html
> 
> This is entirely applicable to streaming too.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message