kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Correct usage of consumer groups
Date Tue, 29 May 2018 18:21:55 GMT
About the docs:

Config `cleanup.policy` states:

> A string that is either "delete" or "compact".
> This string designates the retention policy to
> use on old log segments. The default policy> ("delete") will discard old segments
when their
> retention time or size limit has been reached.> The "compact" setting will enable
> compaction on the topic.

Because deletions happens based on segments, it it clear that some
messages are retained longer, because a segment can only be dropped if
_all_ messages in a segment passed the retention time.

Does this make sense?

Of course, we are always happy to improve the docs. Feel free to do a PR :)


On 5/29/18 3:01 AM, Shantanu Deshmukh wrote:
> In one of my consumer application, I saw that 3 topics with 10 partitions
> each were getting consumed by 5 different consumers having same consumer
> group. And this application is seeing a lot of rebalances. Hence, I was
> wondering about this.
> On Tue, May 29, 2018 at 1:57 PM M. Manna <manmedia@gmail.com> wrote:
>> topic and consumer group have 1-to-many relationship. Each topic partition
>> will have the messages guaranteed to be in order. Consumer rebalance issues
>> can be adjusted based on the backoff and other params. What is exactly your
>> concern regarding consumer group and rebalance?
>> On 29 May 2018 at 08:26, Shantanu Deshmukh <shantanu88d@gmail.com> wrote:
>>> Hello,
>>> Is it wise to use a single consumer group for multiple consumers who
>>> consume from many different topics? Can this lead to frequent rebalance
>>> issues?

View raw message