kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taylor Gautier <tgaut...@tagged.com>
Subject Re: get offsets?
Date Thu, 22 Sep 2011 02:06:35 GMT
I see (and I think I understand why it works this way).

I see that kafka-87 addresses this with a request for having a time based
index, this would be relatively useful, but I also would like to have a way
to go back say 1,000 messages.  Other than walking backwards one segment at
a time, can then scanning forward from there, do you have any suggestions
how this might be done or is it also a feature request?

On Wed, Sep 21, 2011 at 5:14 PM, Joel Koshy <jjkoshy@gmail.com> wrote:

> Hi Taylor,
>
> This is an FAQ, that was asked some time ago as well:
>
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201108.mbox/%3CCAFbh0Q3QgsYXn4kjTTh0zZ9DJZ7J14tM5g24RFJSihFfbqDNyw@mail.gmail.com%3E
>
> GetOffsetShell doesn't return the offset of every message. It returns the
> offset of the first message in every segment file. If you provide a time,
> the offsets you get back are based on the last modified time of the segment
> files.
>
> It may be a good idea to paraphrase the above to be displayed with the
> tool's help message.
>
> Thanks,
>
> Joel
>
> On Wed, Sep 21, 2011 at 4:18 PM, Taylor Gautier <tgautier@tagged.com>
> wrote:
> > Hi,
> >
> > Using Kafka 0.6.
> >
> > I'm trying to use get offsets but it doesn't seem to work as I expect.  I
> > have  a test topic that has some messages in it.  Here's the output of a
> > test client that starts from offset 0 and prints all messages/offsets for
> > the topic:
> >
> > Consumed message:foo offset: 12
> > Consumed message:bar offset: 24
> > Consumed message:foo offset: 36
> > Consumed message:bar offset: 48
> > Consumed message:hello offset: 62
> > Consumed message:world offset: 76
> >
> > Here's a class to print the last n offsets:
> >
> > public class SimpleConsumerDemo
> >
> > {
> >
> >  public static void main(String[] args)
> >
> >  {
> >
> >    SimpleConsumer simpleConsumer = new SimpleConsumer("localhost", 9092,
> > 1000, 1024);
> >
> >
> >    long[] offsets = simpleConsumer.getOffsetsBefore("test", 0, -1L, 1);
> >
> >    for (long l : offsets) {
> >
> >        System.out.println("offset: " + l);
> >
> >    }
> >
> >  }
> >
> > }
> >
> >
> > Running this as above, with 1 offset, yields expected results:
> >
> > ----output-----
> >
> > offset: 76
> >
> >
> > However, asking for 3 offsets yields unexpected results:
> >
> >
> > change to: long[] offsets = simpleConsumer.getOffsetsBefore("test", 0,
> -1L,
> > 3);
> >
> > ----output-----
> >
> > offset: 76
> >
> > offset: 0
> >
> >
> > I expected:
> >
> >
> > ----output-----
> >
> > offset: 76
> >
> > offset: 62
> >
> > offset: 48
> >
> > Any idea why I did not get what I was looking for/what I am doing wrong?
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message