kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Subhash Sriram <subhash.sri...@gmail.com>
Subject Re: Kafka 0.10.1.0 - Question about kafka-consumer-groups.sh
Date Fri, 05 May 2017 21:53:23 GMT
Thanks for confirming, Vahid. Unfortunately, that is not what I am seeing!
I see a consumer actively receiving messages from a topic, but its group
name doesn't show up when I run a list.

I will see if I can reproduce this using a test that I ran today after the
offsets expire. Perhaps I can pause the producers, wait for the existing
group to reach the end of the topic, and then restart the consumers with a
new group name.

Thanks again for your help.
Subhash

On Fri, May 5, 2017 at 5:46 PM, Vahid S Hashemian <vahidhashemian@us.ibm.com
> wrote:

> Hi Subhash,
>
> Yes, it should be listed again in the output, but it should get fresh
> offsets (with `--describe` for example), since the old offsets were
> removed once it became inactive.
>
> --Vahid
>
>
>
>
> From:   Subhash Sriram <subhash.sriram@gmail.com>
> To:     users@kafka.apache.org
> Date:   05/05/2017 02:38 PM
> Subject:        Re: Kafka 0.10.1.0 - Question about
> kafka-consumer-groups.sh
>
>
>
> Hi Vahid,
>
> Thank you very much for your reply! I appreciate the clarification.
> Unfortunately, I didn't really try the command until today. That being
> said, I created a couple of new groups and consumed from a test topic
> today, and they did show up in the list. Maybe I can see what happens with
> them after the offset.retention.minutes has elapsed.
>
> What if a group went inactive, and then became active again with the same
> group name? I would assume that once it turns back on, it would show up in
> the list.
>
> Thanks again,
> Subhash
>
>
> On Fri, May 5, 2017 at 5:30 PM, Vahid S Hashemian
> <vahidhashemian@us.ibm.com
> > wrote:
>
> > Hi Subhash,
> >
> > The broker config that affects group offset retention is
> > "offsets.retention.minutes" (which defaults to 1 day).
> > If the group is inactive (i.e. has no consumer consuming from it) for
> this
> > long its offsets will be removed from the internal topic offset and it
> > will not be listed in the consumer group command output.
> >
> > But the consumer group in your case should be alive, since it did not
> > become inactive.
> >
> > Did the command use to list the group in the output before?
> >
> > --Vahid
> >
> >
> >
> >
> > From:   Subhash Sriram <subhash.sriram@gmail.com>
> > To:     users@kafka.apache.org
> > Date:   05/05/2017 01:43 PM
> > Subject:        Kafka 0.10.1.0 - Question about kafka-consumer-groups.sh
> >
> >
> >
> > Hey everyone,
> >
> > I am a little bit confused about how the kafka-consumer-groups.sh/
> > ConsumerGroupCommand works, and was hoping someone could shed some light
> > on
> > this for me.
> >
> > We are running Kafka 0.10.1.0, and using the new Consumer API with the
> > Confluent.Kafka C# library (v0.9.5) that uses librdkafka. Today, I was
> > trying to get some details on what consumers were running, and their
> > position within a couple of topics, but when I ran the following, I did
> > not
> > see the group in the list.
> >
> > ./bin/kafka-consumer-groups.sh --bootstrap-server [our servers]
> > --new-consumer --list
> >
> > I see a few groups listed, but none of the ones I was expecting to see.
> I
> > saw this in the documentation:
> >
> > *"When using the new consumer API
> > <http://kafka.apache.org/documentation.html#newconsumerapi> (where the
> > broker handles coordination of partition handling and rebalance), the
> > group
> > is deleted when the last committed offset for that group expires"*
> >
> > Is that related to log retention time? If so, is it saying that the
> group
> > will be deleted from the list once the highest committed offset of its
> > group is past its configured log retention time?
> >
> > The issue I am facing is that we have a consumer group that is actively
> > consuming. At one point, I am sure the messages in the topic it is
> > consuming expired, but since then, more messages have been added, and it
> > has been consuming & committing higher offsets. Shouldn't that group
> have
> > come back on the list?
> >
> > Any ideas would be very helpful.
> >
> > Thanks,
> > Subhash
> >
> >
> >
> >
> >
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message