kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sachin Mittal <sjmit...@gmail.com>
Subject Re: What does GetOffsetShell result represent
Date Sun, 18 Dec 2016 06:29:05 GMT
Hi,
Thanks for the lucid explanation.

So say if I run this
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
192.168.73.198:9092 --topic test-stream --time -1 --offsets 6
I get output like:
test-stream:0:81658712,81616045,81299569,80987584,80673168,80364832

This means these are the offset of latest message on these segment files.
So the current segment is holding (81658712 - 81616045) messages = 42667
message
Previous one had (81616045 - 81299569) = 316476 messages
Previous than that (81299569 - 80987584) = 311985 messages
and so on...

and when I run
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
192.168.73.198:9092 --topic test-stream --time -2
I get output like
test-advice-stream:0:69258918

This means that earliest message offset in the segment file is 69258918 and
messages before that have been deleted.

I guess my understating is correct.

Thanks
Sachin




On Sun, Dec 18, 2016 at 6:25 AM, Ewen Cheslack-Postava <ewen@confluent.io>
wrote:

> The tool writes output in the format:
>
> <topic>:<partition>:<comma separated list of valid offsets>
>
> So in the case of your example with --time -1 that returned
> test-window-stream:0:724, it is saying that test-window-stream has
> partition 0 with a valid log segment which has the first offset = 724. Note
> that --time -1 is a special code for "only give the latest segment". For
> the -2 option you saw 0:0 because the earliest segment is still currently
> the first segment created for that topic, i.e. the one starting with offset
> 0 (because your retention policy hasn't kicked in yet). In cases where it
> returned multiple comma separated values, those are multiple starting
> offsets for segments containing offsets before the timestamp you requested.
>
> Note that this is using the old simple consumer approach to querying for
> offsets. That implementation could only tell you about offsets based on
> entire log segments. If you're on a newer version of Kafka, you may be
> interested in the time-based offset query, which you can access in the new
> consumer API and gives you better granularity (
> http://kafka.apache.org/0101/
> javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#
> offsetsForTimes(java.util.Map)).
>
> -Ewen
>
> On Thu, Dec 15, 2016 at 3:52 AM, Sachin Mittal <sjmittal@gmail.com> wrote:
>
> > Folks any explanation for this. Or any link that can help me on that.
> >
> > On Tue, Dec 13, 2016 at 1:00 PM, Sachin Mittal <sjmittal@gmail.com>
> wrote:
> >
> > > Hi,
> > > I have some trouble interpreting the result of GetOffsetShell command.
> > >
> > > Say if I run
> > > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell
> --broker-list
> > > localhost:9092 --topic test-window-stream --time -2
> > > test-window-stream:0:0
> > >
> > > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> > > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> > > test-window-stream --time -1
> > > test-window-stream:0:724
> > >
> > > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell
> --broker-list
> > > localhost:9092 --topic test-window-stream
> > > test-window-stream:0:724
> > >
> > > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> > > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> > > test-window-stream --offset 2
> > > test-window-stream:0:724,0
> > >
> > > So what does 0:0 mean or for that mater 0:724
> > >
> > > Also when I use  -2 (earliest), why does it show 0:0, does this means
> > > offset if at the start of the topic.
> > > Also when I use -1 (latest), it shows 0:724, does it means none of the
> > > records are processed.
> > > However I was having a stream application running which did aggregate
> > > atleast  upto 720 records before I killed it. So why does it not show
> > here?
> > >
> > > Also it is not clear what --offset flag means and when I pass something
> > > greater than 1 I get ,0 appended to the earlier result.
> > >
> > > Thanks
> > > Sachin
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message