incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yu Zheng <bearzheng2...@gmail.com>
Subject Re: S4 Load Balancing Mechanism
Date Fri, 12 Oct 2012 01:57:38 GMT
Hi Aimee,
Thank you for your prompt reply.

Best regards,
Yu
On Oct 12, 2012, at 9:55 AM, Aimee Cheng <chengsj.luo@gmail.com> wrote:

> I can answer your second question: no. In current S4 piper, when there are no standby
nodes, if one node crushes, the remaining nodes will not pick up the task and still focus
on their own tasks.
> 
> -Aimee
>  
> On Oct 12, 2012, at 9:36 AM, Frank Zheng wrote:
> 
>> Hi,
>> 
>> Thanks for your reply.
>> So the dynamic load balancing is not implemented yet, right?
>> In case of static load balancing, if there are two working PE node without any standby
nodes, 
>> when one fails, does the workload for the remaining working PE increase?
>> 
>> Besides, can zeno prototype do dynamic load balancing? 
>> 
>> Thanks.
>> 
>> On Thu, Oct 11, 2012 at 7:00 PM, Matthieu Morel <mmorel@apache.org> wrote:
>> On 10/11/12 12:35 PM, Daniel Gómez Ferro wrote:
>> 
>> 
>> On Thu Oct 11 05:52:01 2012, Frank Zheng wrote:
>> Hi All,
>> 
>> I want to figure out the mechanism of load balancing in S4.
>> In the paper and web site, it says ZooKeeper takes this
>> responsibility, but the description is not very clear.
>> I want to figure out that how ZooKeeper does load balancing.
>> For example, if there are two working PE nodes, and 120 messages per
>> second. The adapter distributes these messages evenly, 60 per PE node,
>> right?
>> When one PE node is added into the cluster, does it mean ZooKeeper
>> balances the messages, 40 per PE node?
>> And how does ZooKeeper decide which message to be distributed to which
>> node?
>> 
>> Hi,
>> 
>> Currently load balancing is static. When you define your cluster you
>> specify the number of tasks (say, 2) which won't change for this
>> cluster. If you have more than 2 nodes, the rest will be backup nodes.
>> In case one of the serving nodes fails, a backup node will pick up the
>> orphaned task and start serving requests, but the load will always be
>> spread among the 2 tasks that you defined at the beginning.
>> 
>> Having dynamic load balancing would be really cool, but I don't know if
>> there are currently any efforts in that direction.
>> 
>> There is the zeno prototype https://github.com/s4/zeno , which is actually independent
of S4. It also relies on ZooKeeper for metadata about partitions.
>> 
>> Matthieu
>> 
>> 
>> 
>> -- 
>> Sincerely,
>> Zheng Yu
>> Mobile:  (852) 60670059
>> Email:    bearzheng2011@gmail.com
>> 
>> 
>> 
> 


Mime
View raw message