qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Fox <tim....@jboss.com>
Subject Re: clustering
Date Fri, 22 Dec 2006 19:21:48 GMT

lichtner wrote:
> On Fri, 22 Dec 2006, Tim Fox wrote:
>> Personally I would never use a total ordering protocol (e.g. totem) to
>> implement clustering for messaging.
>> Naive clustering approaches often try to replicate the queue state
>> machine across different nodes, and require some kind of total ordering
>> to ensure the states are consistent.
>> You'll find this approach doesn't scale, since every replica in the
>> cluster has to receive all updates from every other node. Therefore the
>> amount of load on a particular node increases linearly with the number
>> of nodes in the cluster.
> But I am proposing that all queues be available on all nodes. So all the
> nodes do need to see all the messages.

That's only true if you want to maintain exact copies of the queues on 
each node. If you want to replicate for high availability that might be 

However, if you're intention is to be able to scale your system by 
adding nodes to the cluster then maintaining exact replicas is not going 
to help, since as we've both tacitly agreed each node would receive the 
traffic from all other nodes, so you make no gains in adding extra nodes 
to the cluster. (I.e. clustering is pointless)

This boils down to the fact that the work in adding/removing an entry 
from a strict FIFO queue is not inherently parallelizable (see Amdahl's 

So how do you solve this problem? One way (and this is how several of 
the popular messaging systems do it) do it is to relax the strict 
FIFOness of the queue.

For many messaging applications/flavours, e.g. JMS, strict FIFO is not 
necessary, it is only necessary to maintain order of messages produced 
from a session. Therefore you can implement the "logical" clustered 
queue, as a set of quasi-independent partial queues - one on each node 
of the cluster. When you are attached at a particular node and sending a 
message to the clustered queue, the particular partial queue it ends up 
in is determined by a routing policy - typically you would always favour 
the local queue to avoid extra network traffic. Consumers can be 
attached at any node and consume from the partial queue. Then you 
implement a "message redistribution" algorithm on top of this which 
pulls messages from one partial quuee to another to avoid starvation of 
consumers on a particular partial queue, or to spread load due to 
consumers faster or slower on different nodes.

Theoretically this has linear scalability.


Tim Fox
JBoss Messaging Technical Lead
JBoss - a division of Red Hat
T: +44 2088006768
M: +44 7957983205
E: tim.fox@jboss.com tim.fox@redhat.com

View raw message