storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Mahadevan <ar...@apache.org>
Subject Re: implmement state management with Apache Ignite
Date Thu, 16 Feb 2017 05:32:27 GMT
Hi Shawn,

 

> After study, We will implement IgniteKeyValueState and IgniteStateProvider. 

 

Sure, it will be great if you can implement and contribute it to the storm community.

 

>Could storm add support for ignite? 

>porting RedisKeyValueState to IgniteKeyValueState is rather easy.

 

Once you have the implementation, you can raise a pull request and it will get reviewed and
merged to the storm codebase. In case you are not sure about the process, check this out -
https://github.com/apache/storm/blob/master/DEVELOPER.md

 

Thanks,

Arun

 

 

From: "shawn.du" <shawn.du@neulion.com.cn>
Reply-To: "user@storm.apache.org" <user@storm.apache.org>
Date: Thursday, February 16, 2017 at 7:30 AM
To: user <user@storm.apache.org>
Cc: user <user@storm.apache.org>
Subject: Re: implmement state management with Apache Ignite

 

Hi,

 

After study, We will implement IgniteKeyValueState and IgniteStateProvider. 

I am a ignite user, I think Ignite has more wonderful features than redis.

Could storm add support for ignite? 

porting RedisKeyValueState to IgniteKeyValueState is rather easy.

 

Thanks

Shawn

 

On 02/7/2017 14:39,shawn.du<shawn.du@neulion.com.cn> wrote: 

Hi Arun,

 

Thanks your reply. I have read the document your mentioned. 

For current storm 1.0.2 only provide a simple KeyValueState interface,  We are still considering
whether it can fulfill our requirements.

 

Thanks

Shawn

 

On 02/7/2017 14:08,Arun Mahadevan<arunm@apache.org> wrote: 

> #1 when a worker is died or killed by manually, storm framework will restart this worker,
is there a ID which doesn't change for the new worker and the died worker?  if there is, how
to get it?

Task Id does not change. The worker can be restarted in the same or different host and the
tasks would be reassigned.

 

> #2 when a worker is died, is there a case storm framework can't start a new worker? In
this case, how about the grouped message handled by this worker, it will auto take over by
other workers?   

As long as the state is per task it can be handled by other worker.

 

> #3 suppose we have started a topology with 3 workers, rebalance topology can change the
worker numbers?

Rebalance can change the number of workers, but the overall number of tasks remains the same.
Only that the task assignment to workers could change.

 

Not sure if you are building something different, but storm already has basic support for
stateful bolts (see https://github.com/apache/storm/blob/master/docs/State-checkpointing.md).

Thanks,

Arun

 

 

From: "shawn.du" <shawn.du@neulion.com.cn>
Reply-To: "user@storm.apache.org" <user@storm.apache.org>
Date: Tuesday, February 7, 2017 at 11:25 AM
To: user <user@storm.apache.org>
Subject: implmement state management with Apache Ignite

 

Hi community,

 

Recently we are trying implement state management using apache ignite.

Before do the real work, I have several questions, please help.

 

#1 when a worker is died or killed by manually, storm framework will restart this worker,
is there a ID which doesn't change for the new worker and the died worker?  if there is, how
to get it?

#2 when a worker is died, is there a case storm framework can't start a new worker? In this
case, how about the grouped message handled by this worker, it will auto take over by other
workers?   

#3 suppose we have started a topology with 3 workers, rebalance topology can change the worker
numbers?

 

Thanks

Shawn

 


Mime
View raw message