qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Conway <acon...@redhat.com>
Subject Re: Dynamic balancing of queues.
Date Mon, 11 Sep 2006 14:37:34 GMT
On Mon, 2006-09-11 at 14:37 +0100, Gordon Sim wrote:
> Why do we need extra queues to scale up? 

For a single broker you don't, the broker should be optimized to take
full advantage of local CPUs etc. with a single queue.

I'm thinking about the large scale deployment where you need to
distribute load across multiple hosts (possibly on different networks)
because either the CPUs on the exchange host or it's network
connectivity are a bottleneck. 

> > So let the exchange act as a consumer and *remove* messages from
> > over-full queues to re-queue them on under-full/empty ones. Now we have
> > a full dynamic balance in both growing and shrinking phases
> 
> This complicates the exchange though as it now presumably needs to 
> periodically monitor the queue lengths (i.e. it becomes an active entity 
> rather than just reacting to publications routed through it). Maybe an 
> entirely separate re-balancing component would be cleaner?

Maybe, need to think about that. My intuition is that the broker can do
a better job of this because it's a single place to keep the statistics,
but a distributed cleanup component might have advantages if network
bandwidth at the broker is the bottlneck.

> I'm not really sure I understand the root problem here. i.e. why do we 
> want multiple queues of the same (or similar) length?

Because you want to keep all the the consumers busy all the time. If you
have 10 queues and 5 of them empty out because they have fast consumers,
and you don't have any new work to put on them, then you don't want to
leave them idle while the other 5 slow consumers chew through the
remaining jobs.

The ideal solution to this is of course to have only 1 queue and let
each consumer take items at its own pace - that naturally balances
itself. 

Trying to balance multiple queues only makes sense if there's a resource
problem with a everyone talking to a single queue - not enough memory,
not enough open file descriptors, performance degrades due to memory
requirements, network topology/firewalls etc. You can imagine situations
where a single broker with 1,000,000 consumers might not perform as well
as a federation of 1001 brokers each with 1000 consumers.

That said need to work hard optimizing the broker so that the single
queue solution can scale as far as possible before we get into more
complicated federations and the like. We should also look at some real
data before assuming that such federation will solve a real-world
problem, this stuff doesn't always work out the way you think it will!

 


Mime
View raw message