kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Hauser <ewhau...@gmail.com>
Subject Re: RPC on top of kafka
Date Fri, 30 Sep 2011 00:28:59 GMT
There are a things you might want to consider before using Kafka for
your use case:

1. Kafka producers currently do not get an acknowledgement, which may
or may not be a big deal for you
2. No messages will show up at your consumers until they hit disk (and
Kafka will keep all your request/response data on disk until the
retention period is reached).
3. Kafka is not really designed for low latency messaging.    You can
toggle some of the flush settings per topic, but you won't get the
same performance as something that is queuing in memory only.

On Thu, Sep 29, 2011 at 11:40 AM, Pierre-Yves ritschard
<pyr@smallrivers.com> wrote:
> Hi,
>
> It would be short lived consumer sessions: setup consumer + consumer
> group, produce to worker, accept response, teardown consumer. So it
> would stay well beneath the maxopenfile threshold.
>
> What I was worried about is the potential overhead of creating and
> tearing down consumer groups since it generates zookeeper interaction.
>
>  - pyr
>
> On Thu, 29 Sep 2011 08:35:33 -0700
> Jun Rao <junrao@gmail.com> wrote:
>
>> Pierres-Yves,
>>
>> Since each consumer maintains its own state, there is relatively
>> little overhead per consumer on the broker. However, each consumer
>> has to maintain a socket connection to a broker, you can't have
>> infinite number of consumers. How many concurrent consumer groups do
>> you plan to have?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Sep 29, 2011 at 7:44 AM, Pierre-Yves ritschard
>> <pyr@smallrivers.com>wrote:
>>
>> > Hi,
>> >
>> > One of the standard use cases for messaging queues is to implement
>> > some sort of distributed RPC.
>> >
>> > Let's pretend you have a bunch of worker processes, all registered
>> > in the same consumer group (to make sure only one is done at a
>> > time) and which are registered on a queue named to represent the
>> > task they know how to do.
>> >
>> > RPC could then be implemented by a dynamic consumer group creation
>> > and registration on the client side, then sending a message
>> > containing the created id in the payload, so that the worker
>> > process knows who to respond to.
>> >
>> > What I'm worried about is the overhead of registering new consumer
>> > groups on the fly, would that put an anormal burden on the brokers ?
>> >
>> >  - pyr
>> >
>
>

Mime
View raw message