kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre-Yves ritschard <...@smallrivers.com>
Subject Re: RPC on top of kafka
Date Thu, 29 Sep 2011 15:40:29 GMT
Hi,

It would be short lived consumer sessions: setup consumer + consumer
group, produce to worker, accept response, teardown consumer. So it
would stay well beneath the maxopenfile threshold.

What I was worried about is the potential overhead of creating and
tearing down consumer groups since it generates zookeeper interaction.

  - pyr

On Thu, 29 Sep 2011 08:35:33 -0700
Jun Rao <junrao@gmail.com> wrote:

> Pierres-Yves,
> 
> Since each consumer maintains its own state, there is relatively
> little overhead per consumer on the broker. However, each consumer
> has to maintain a socket connection to a broker, you can't have
> infinite number of consumers. How many concurrent consumer groups do
> you plan to have?
> 
> Thanks,
> 
> Jun
> 
> On Thu, Sep 29, 2011 at 7:44 AM, Pierre-Yves ritschard
> <pyr@smallrivers.com>wrote:
> 
> > Hi,
> >
> > One of the standard use cases for messaging queues is to implement
> > some sort of distributed RPC.
> >
> > Let's pretend you have a bunch of worker processes, all registered
> > in the same consumer group (to make sure only one is done at a
> > time) and which are registered on a queue named to represent the
> > task they know how to do.
> >
> > RPC could then be implemented by a dynamic consumer group creation
> > and registration on the client side, then sending a message
> > containing the created id in the payload, so that the worker
> > process knows who to respond to.
> >
> > What I'm worried about is the overhead of registering new consumer
> > groups on the fly, would that put an anormal burden on the brokers ?
> >
> >  - pyr
> >


Mime
View raw message