kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: What does GetOffsetShell result represent
Date Sun, 18 Dec 2016 00:55:03 GMT
The tool writes output in the format:

<topic>:<partition>:<comma separated list of valid offsets>

So in the case of your example with --time -1 that returned
test-window-stream:0:724, it is saying that test-window-stream has
partition 0 with a valid log segment which has the first offset = 724. Note
that --time -1 is a special code for "only give the latest segment". For
the -2 option you saw 0:0 because the earliest segment is still currently
the first segment created for that topic, i.e. the one starting with offset
0 (because your retention policy hasn't kicked in yet). In cases where it
returned multiple comma separated values, those are multiple starting
offsets for segments containing offsets before the timestamp you requested.

Note that this is using the old simple consumer approach to querying for
offsets. That implementation could only tell you about offsets based on
entire log segments. If you're on a newer version of Kafka, you may be
interested in the time-based offset query, which you can access in the new
consumer API and gives you better granularity (http://kafka.apache.org/0101/
javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#
offsetsForTimes(java.util.Map)).

-Ewen

On Thu, Dec 15, 2016 at 3:52 AM, Sachin Mittal <sjmittal@gmail.com> wrote:

> Folks any explanation for this. Or any link that can help me on that.
>
> On Tue, Dec 13, 2016 at 1:00 PM, Sachin Mittal <sjmittal@gmail.com> wrote:
>
> > Hi,
> > I have some trouble interpreting the result of GetOffsetShell command.
> >
> > Say if I run
> > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell --broker-list
> > localhost:9092 --topic test-window-stream --time -2
> > test-window-stream:0:0
> >
> > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> > test-window-stream --time -1
> > test-window-stream:0:724
> >
> > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell --broker-list
> > localhost:9092 --topic test-window-stream
> > test-window-stream:0:724
> >
> > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> > test-window-stream --offset 2
> > test-window-stream:0:724,0
> >
> > So what does 0:0 mean or for that mater 0:724
> >
> > Also when I use  -2 (earliest), why does it show 0:0, does this means
> > offset if at the start of the topic.
> > Also when I use -1 (latest), it shows 0:724, does it means none of the
> > records are processed.
> > However I was having a stream application running which did aggregate
> > atleast  upto 720 records before I killed it. So why does it not show
> here?
> >
> > Also it is not clear what --offset flag means and when I pass something
> > greater than 1 I get ,0 appended to the earlier result.
> >
> > Thanks
> > Sachin
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message