kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marina <ppi...@yahoo.com.INVALID>
Subject Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Date Thu, 03 Dec 2015 18:55:40 GMT
Hi, Guozhang,
Yes, I can see this topic and partitions in ZK:

ls /brokers/topics/__consumer_offsets
[partitions]
ls /brokers/topics/__consumer_offsets/partitions
[44, 45, 46, 47, 48, 49, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 0, 1, 2, 3, 4, 5, 6, 7, 8,
9, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,
42, 43]


example of one partition state:
ls /brokers/topics/__consumer_offsets/partitions/44
[state]
ls /brokers/topics/__consumer_offsets/partitions/44/state
[]


anything else you think I could check?

thanks!
Marina



----- Original Message -----
From: Guozhang Wang <wangguoz@gmail.com>
To: "users@kafka.apache.org" <users@kafka.apache.org>; Marina <ppine7@yahoo.com>
Sent: Thursday, December 3, 2015 12:05 PM
Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?

Marina,

To check if the topic does exist in Kafka (i.e. offsets are stored in Kafka
instead of in ZK) you can check this path in ZK:

/brokers/topics/__consumer_offsets

By default this topic should have 50 partitions.

Guozhang



On Thu, Dec 3, 2015 at 6:22 AM, Marina <ppine7@yahoo.com.invalid> wrote:

> Hi, Jason,
>
> I tried the same command both with specifying a formatter and without -
> same result:
>
> => /opt/kafka/bin/kafka-console-consumer.sh --formatter
> kafka.server.OffsetManager\$OffsetsMessageFormatter --consumer.config
> /tmp/consumer.properties --topic __consumer_offsets --zookeeper
> localhost:2181 --from-beginning
>
> ^CConsumed 0 messages
>
>
> When you are saying to make sure Consumers are using Kafka instead of ZK
> storage for offset... - what is the best way to do that? I thought checking
> the content via ZK shell - at this path:
> /consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>
>
> would be a definitive confirmation (there are no values stored there). Is
> it true?
>
> I feel it is some small stupid mistake I'm making, since it seems to work
> fine for others - drives me crazy :)
>
> thanks!!
> Marina
>
>
>
> ----- Original Message -----
> From: Jason Gustafson <jason@confluent.io>
> To: users@kafka.apache.org
> Sent: Wednesday, December 2, 2015 2:33 PM
> Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
>
> Hey Marina,
>
> My mistake, I see you're using 0.8.2.1. Are you also providing the
> formatter argument when using console-consumer.sh? Perhaps something like
> this:
>
> bin/kafka-console-consumer.sh --formatter
> kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
> localhost:2181 --topic __consumer_offsets --from-beginning
>
> You may also want to confirm that your consumers are using Kafka instead of
> Zookeeper for offset storage. If you still don't see anything, we can
> always look into the partition data directly...
>
> -Jason
>
>
> On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <jason@confluent.io>
> wrote:
>
> > Looks like you need to use a different MessageFormatter class, since it
> > was renamed in 0.9. Instead use something like
> > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
> >
> > -Jason
> >
> > On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> > d.muralidharan@yottaa.com> wrote:
> >
> >> I have this same problem . Can someone help ?
> >>
> >> --Dhyan
> >>
> >> On Wed, Nov 25, 2015 at 3:31 PM, Marina <ppine7@yahoo.com.invalid>
> wrote:
> >>
> >> > Hello,
> >> >
> >> > I'm trying to find out which offsets my current High-Level consumers
> are
> >> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set in
> >> the
> >> > server.properties of Kafka - which, I think, means that offsets are
> >> stored
> >> > in Kafka. (I also verified that no offsets are stored in Zookeeper by
> >> > checking this path in the Zk shell:
> >> >
> >>
> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
> >> > )
> >> >
> >> > I tried to listen to the **__consumer_offsets** topic to see which
> >> > consumer saves what value of offsets, but it did not work...
> >> >
> >> > I tried the following:
> >> >
> >> > created a config file for console consumer as following:
> >> >
> >> >
> >> >     => more kafka_offset_consumer.config
> >> >
> >> >      exclude.internal.topics=false`
> >> >
> >> > and tried to versions of the console consumer scripts:
> >> >
> >> >     #1:
> >> >     bin/kafka-console-consumer.sh --consumer.config
> >> > kafka_offset_consumer.config --topic __consumer_offsets --zookeeper
> >> > localhost:2181
> >> >
> >> >     #2
> >> >     ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets
> >> > --partition 0 --broker-list localhost:9092 --formatter
> >> > "kafka.server.OffsetManager\$OffsetsMessageFormatter"
> --consumer.config
> >> > kafka_offset_consumer.config
> >> >
> >> >
> >> > Neither worked - it just sits there but does not print anything, even
> >> > though the consumers are actively consuming/saving offsets.
> >> >
> >> > Am I missing some other configuration/properties ?
> >> >
> >> > thanks!
> >> >
> >> > Marina
> >> >
> >> > I have also posted this question on StackOverflow:
> >> >
> >> >
> >> >
> >>
> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
> >> >
> >>
> >
> >
>



-- 
-- Guozhang

Mime
View raw message