kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Sites <Eric.Si...@threattrack.com>
Subject Very low volume topic
Date Wed, 14 Aug 2013 01:47:10 GMT
Hello everyone,

I have a very low volume topic that has 2 consumers in the same group. How do I get each consumer
to only consume 1 message at a time and if the the first consumer is busy get the other consumer
to consume the message?

Currently what I am doing is:

First consumer connects to Kafka waits for 300 milliseconds then disconnects, waits for 10
seconds, then reconnects to see if there is a waiting message.

The messages kick off a long task on each server, each server can handle multiple tasks up
to a limit so first I am trying to balance the tasks across multiple servers and if they are
maxed out don't consume any messages.

This will give the other server or servers a chances to pickup a message and do the task.

I would not disconnect if I can ensure I don't have messages waiting in the queue for a server
to consume them without the other servers being able to see them.

Thanks for the help...

Eric Sites

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message