storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anshu shukla <anshushuk...@gmail.com>
Subject Re: implmement state management with Apache Ignite
Date Tue, 07 Feb 2017 10:58:17 GMT
Hey ,

Nice discussion, but as per my knowledge, every *task have its own state*
which gets lost on rebalance operation(this is the reason why tasks can not
be increased at run time).
Mentioned @ https://github.com/apache/storm/pull/1874

So I have 2 questions -
*1- How this state correlates to <key,val> pair in these stateful tasks
mentioned in the links. *Is it separate for every task ?
*2- Whats is the impact fo rebalance operation on these bolts.*


On Tue, Feb 7, 2017 at 11:40 AM, Arun Mahadevan <arunm@apache.org> wrote:

> > #1 when a worker is died or killed by manually, storm framework will
> restart this worker, is there a ID which doesn't change for the new worker
> and the died worker?  if there is, how to get it?
>
> Task Id does not change. The worker can be restarted in the same or
> different host and the tasks would be reassigned.
>
>
>
> > #2 when a worker is died, is there a case storm framework can't start a
> new worker? In this case, how about the grouped message handled by this
> worker, it will auto take over by other workers?
>
> As long as the state is per task it can be handled by other worker.
>
>
>
> > #3 suppose we have started a topology with 3 workers, rebalance topology
> can change the worker numbers?
>
> Rebalance can change the number of workers, but the overall number of
> tasks remains the same. Only that the task assignment to workers could
> change.
>
>
>
> Not sure if you are building something different, but storm already has
> basic support for stateful bolts (see https://github.com/apache/
> storm/blob/master/docs/State-checkpointing.md).
>
> Thanks,
>
> Arun
>
>
>
>
>
> *From: *"shawn.du" <shawn.du@neulion.com.cn>
> *Reply-To: *"user@storm.apache.org" <user@storm.apache.org>
> *Date: *Tuesday, February 7, 2017 at 11:25 AM
> *To: *user <user@storm.apache.org>
> *Subject: *implmement state management with Apache Ignite
>
>
>
> Hi community,
>
>
>
> Recently we are trying implement state management using apache ignite.
>
> Before do the real work, I have several questions, please help.
>
>
>
> #1 when a worker is died or killed by manually, storm framework will
> restart this worker, is there a ID which doesn't change for the new worker
> and the died worker?  if there is, how to get it?
>
> #2 when a worker is died, is there a case storm framework can't start a
> new worker? In this case, how about the grouped message handled by this
> worker, it will auto take over by other workers?
>
> #3 suppose we have started a topology with 3 workers, rebalance topology
> can change the worker numbers?
>
>
>
> Thanks
>
> Shawn
>
>
>



-- 
Thanks & Regards,
Anshu Shukla

Mime
View raw message