kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Manna" <manme...@gmail.com>
Subject Best Practice for Consumer Liveliness and avoid frequent rebalancing
Date Thu, 31 May 2018 12:29:47 GMT
Hello,

We are trying to move from single partition to multi-partition approach for
our topics. The purpose is:

1) Each production/testbed server will have a non-Daemon thread (consumer)
running.
2) It will consume messages, commit offset (manual), and determine next
steps if commit fails, app fails etc.
3) Ideally, 1 partition per server (consumer). If rebalance occurs, first
(lexi ordered) server will end up having additional partition(s).

As I previously understood, and also read Consumer article by Jason
Gustafson
<https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/>,
we should always close consumers for resource optimisation. But departing
from a consumer group means that a rebalance will occur. In our case, we
would like every consumer to be alive (and sleep for a while) but still
send heartbeat so that rebalancing effort is saved. But we re worried
whether this might cause memory leak in our application.

In other words, if we don't restart the servers (shutdown hook), we would
like to avoid invoking KafkaConsumer#close().

Has anyone got similar use case that they can share with us? We are simply
interested to know whether this is a "use-case" scenario or not a good
practice to keep consumers alive.

Any suggestion/help is appreciated.

Regards,

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message