kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Very low volume topic
Date Wed, 14 Aug 2013 04:27:12 GMT
Not sure if there is an easy way to do what you want. One thing I can think
of is to reduce queuedchunks.max and fetch.size to reduce the # of queued
messages.

Thanks,

Jun


On Tue, Aug 13, 2013 at 6:47 PM, Eric Sites <Eric.Sites@threattrack.com>wrote:

> Hello everyone,
>
> I have a very low volume topic that has 2 consumers in the same group. How
> do I get each consumer to only consume 1 message at a time and if the the
> first consumer is busy get the other consumer to consume the message?
>
> Currently what I am doing is:
>
> First consumer connects to Kafka waits for 300 milliseconds then
> disconnects, waits for 10 seconds, then reconnects to see if there is a
> waiting message.
>
> The messages kick off a long task on each server, each server can handle
> multiple tasks up to a limit so first I am trying to balance the tasks
> across multiple servers and if they are maxed out don't consume any
> messages.
>
> This will give the other server or servers a chances to pickup a message
> and do the task.
>
> I would not disconnect if I can ensure I don't have messages waiting in
> the queue for a server to consume them without the other servers being able
> to see them.
>
> Thanks for the help...
>
> Cheers,
> Eric Sites
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message