kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marina <ppi...@yahoo.com.INVALID>
Subject Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?
Date Thu, 03 Dec 2015 14:22:46 GMT
Hi, Jason,

I tried the same command both with specifying a formatter and without - same result:

=> /opt/kafka/bin/kafka-console-consumer.sh --formatter kafka.server.OffsetManager\$OffsetsMessageFormatter
--consumer.config /tmp/consumer.properties --topic __consumer_offsets --zookeeper localhost:2181
--from-beginning

^CConsumed 0 messages


When you are saying to make sure Consumers are using Kafka instead of ZK storage for offset...
- what is the best way to do that? I thought checking the content via ZK shell - at this path:
/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>

would be a definitive confirmation (there are no values stored there). Is it true?

I feel it is some small stupid mistake I'm making, since it seems to work fine for others
- drives me crazy :)

thanks!!
Marina



----- Original Message -----
From: Jason Gustafson <jason@confluent.io>
To: users@kafka.apache.org
Sent: Wednesday, December 2, 2015 2:33 PM
Subject: Re: Kafka 0.8.2.1 - how to read from __consumer_offsets topic?

Hey Marina,

My mistake, I see you're using 0.8.2.1. Are you also providing the
formatter argument when using console-consumer.sh? Perhaps something like
this:

bin/kafka-console-consumer.sh --formatter
kafka.server.OffsetManager\$OffsetsMessageFormatter --zookeeper
localhost:2181 --topic __consumer_offsets --from-beginning

You may also want to confirm that your consumers are using Kafka instead of
Zookeeper for offset storage. If you still don't see anything, we can
always look into the partition data directly...

-Jason


On Wed, Dec 2, 2015 at 11:13 AM, Jason Gustafson <jason@confluent.io> wrote:

> Looks like you need to use a different MessageFormatter class, since it
> was renamed in 0.9. Instead use something like
> "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter".
>
> -Jason
>
> On Wed, Dec 2, 2015 at 10:57 AM, Dhyan Muralidharan <
> d.muralidharan@yottaa.com> wrote:
>
>> I have this same problem . Can someone help ?
>>
>> --Dhyan
>>
>> On Wed, Nov 25, 2015 at 3:31 PM, Marina <ppine7@yahoo.com.invalid> wrote:
>>
>> > Hello,
>> >
>> > I'm trying to find out which offsets my current High-Level consumers are
>> > working off. I use Kafka 0.8.2.1, with **no** "offset.storage" set in
>> the
>> > server.properties of Kafka - which, I think, means that offsets are
>> stored
>> > in Kafka. (I also verified that no offsets are stored in Zookeeper by
>> > checking this path in the Zk shell:
>> >
>> **/consumers/<consumer_group_name>/offsets/<topic_name>/<partition_number>**
>> > )
>> >
>> > I tried to listen to the **__consumer_offsets** topic to see which
>> > consumer saves what value of offsets, but it did not work...
>> >
>> > I tried the following:
>> >
>> > created a config file for console consumer as following:
>> >
>> >
>> >     => more kafka_offset_consumer.config
>> >
>> >      exclude.internal.topics=false`
>> >
>> > and tried to versions of the console consumer scripts:
>> >
>> >     #1:
>> >     bin/kafka-console-consumer.sh --consumer.config
>> > kafka_offset_consumer.config --topic __consumer_offsets --zookeeper
>> > localhost:2181
>> >
>> >     #2
>> >     ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets
>> > --partition 0 --broker-list localhost:9092 --formatter
>> > "kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config
>> > kafka_offset_consumer.config
>> >
>> >
>> > Neither worked - it just sits there but does not print anything, even
>> > though the consumers are actively consuming/saving offsets.
>> >
>> > Am I missing some other configuration/properties ?
>> >
>> > thanks!
>> >
>> > Marina
>> >
>> > I have also posted this question on StackOverflow:
>> >
>> >
>> >
>> http://stackoverflow.com/questions/33925866/kafka-0-8-2-1-how-to-read-from-consumer-offsets-topic
>> >
>>
>
>

Mime
View raw message