qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Conway <acon...@redhat.com>
Subject Re: Proposed QIP: Support for Message Grouping in the broker.
Date Wed, 29 Jun 2011 10:32:23 GMT
On 06/29/2011 11:16 AM, Gordon Sim wrote:
> On 06/29/2011 11:08 AM, Alan Conway wrote:
>> Gordon's mode 1 (stick to a consumer only as long as it has unacked
>> messages) doesn't really do this: the only time it allows load to shift
>> is when the queue is idle long enough for the consumer to ack all its
>> pre-fetched messages. But if the queue is idle then we're not under
>> heavy load which is where load balancing is most important.
> That's not necessarily true. The queue could have plenty of messages on it and
> not be idle, but not have messages in a particular group.

Good point!

> Where I think this mode may make load balancing more adaptive is where there is
> variability in the groups and the processing times of messages.
> For example, consumer 1 gets assigned group 1 and processes 1 available message
> and then accepts it, there are no more messages at present for that group. It
> then gets messages sent for groups 3, 5 and 6 say. If those groups now get a lot
> of messages, consumer 1 may become quite busy. If another set of messages for
> group 1 are published, why not allow them to be processed by a different
> consumer. Assuming of course that the requirement is simply in-order processing.
> I'm certainly not claiming this is perfect. However it seems to me that it is
> not necessary in all cases to maintain the stickiness for the lifetime of the
> consumer.

Indeed, if load balancing over long-lived groups is the primary concern (e.g. 
stock symbols as groups) then it is important that groups do not stick to 
consumers for their lifetime.

>> This makes me think there is a case for a 3rd mode of stickiness:
>> non-stick mode. In this mode we randomly chose the next consumer when
>> the current consumer acks. Clearly this will lower the possible msg/sec
>> throughput since you are effectively doing synchronous messaging.
>> However it may still be relevant where the msg/sec throughput is low
>> anyway but the need for fair load balancing is high - e.g. very large
>> messages or messages that require a long time to process. In that case
>> the extra time added by synchronous messages may be negligible, but the
>> automatic distribution of load is very valuable.
> Why not just set the prefetch on the consumers to 0 or 1, and have the consumer
> accept each message as it is processed. That would seem to achieve the same
> thing and is exactly what you would do to improve the fairness of distribution
> without message groups (again at the expense of throughput).

Excellent point. My non-stick mode is just your mode 2 + prefetch = 0. Very neat.

Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

View raw message