qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Conway <acon...@redhat.com>
Subject Re: Proposed QIP: Support for Message Grouping in the broker.
Date Wed, 29 Jun 2011 10:08:31 GMT
On 06/29/2011 10:37 AM, Marnie McCormack wrote:
> Some useful reading links, for the concept in JMS terms/other
> implementations (which users might expect us to support/be close ot
> semantically). I'm not saying any of these approaches are exactly correct,
> but that the logical behaviour they support is similar.
> http://activemq.apache.org/message-groups.html
> http://www.ibm.com/developerworks/websphere/library/techarticles/0602_currie/0602_currie.html
> http://forum.springsource.org/showthread.php?48798-Spring-2.5.1-JMS-1.1-and-distributed-jobs-implementation

The activemq document gives a good clear description. They are implementing 
Gordon's mode 2: stick to a consumer for as long as it lives.

They also raise an interesting point:

"If you have existing messages in the broker and add consumers at a later stage, 
it is a good idea to delay message dispatch start until all consumers are 
present (or at least to give enough time for them to subscribe). If you don't do 
that the first consumer will probably acquire all message groups and all 
messages will be dispatched to it. You can achieve this by using 
consumersBeforeDispatchStarts and timeBeforeDispatchStarts destination policies. "

In other words you would like to be able to shift load from a busy consumer to a 
less busy one, or have a random distribution of load that will even out on average.

Gordon's mode 1 (stick to a consumer only as long as it has unacked messages) 
doesn't really do this: the only time it allows load to shift is when the queue 
is idle long enough for the consumer to ack all its pre-fetched messages. But if 
the queue is idle then we're not under heavy load which is where load balancing 
is most important.

This makes me think there is a case for a 3rd mode of stickiness: non-stick 
mode. In this mode we randomly chose the next consumer when the current consumer 
acks. Clearly this will lower the possible msg/sec throughput since you are 
effectively doing synchronous messaging. However it may still be relevant where 
the msg/sec throughput is low anyway but the need for fair load balancing is 
high - e.g. very large messages or messages that require a long time to process. 
In that case the extra time added by synchronous messages may be negligible, but 
the automatic distribution of load is very valuable.

Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

View raw message