kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip O'Toole" <phi...@loggly.com>
Subject Re: Very low volume topic
Date Wed, 14 Aug 2013 01:57:05 GMT
My experience is solely with 0.72. More inline.

On Tue, Aug 13, 2013 at 6:47 PM, Eric Sites <Eric.Sites@threattrack.com> wrote:
> Hello everyone,
> I have a very low volume topic that has 2 consumers in the same group. How do I get each
consumer to only consume 1 message at a time and if the the first consumer is busy get the
other consumer to consume the message?

You can't, not if you only have one partition. Each consumer is
dedicated to a single partition. Unless you deliberately tear down the
consumer and let another take over that partition (if you are using
the high-level consumer).

> Currently what I am doing is:
> First consumer connects to Kafka waits for 300 milliseconds then disconnects, waits for
10 seconds, then reconnects to see if there is a waiting message.

I don't think you need to do this. The high-level has a API that
allows you to set this timeout (I think).

> The messages kick off a long task on each server, each server can handle multiple tasks
up to a limit so first I am trying to balance the tasks across multiple servers and if they
are maxed out don't consume any messages.
> This will give the other server or servers a chances to pickup a message and do the task.
> I would not disconnect if I can ensure I don't have messages waiting in the queue for
a server to consume them without the other servers being able to see them.

I think a better design would be to have a basic consumer that drains
the topic and hands jobs to the set of available workers. *Those*
workers perform the long-running job. Only if there are no available
workers does the consumer block. You may be trying to do too much in
the consumer.

> Thanks for the help...
> Cheers,
> Eric Sites

View raw message