kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Palino <tpal...@gmail.com>
Subject Re: Last offset in all partitions
Date Wed, 06 Jul 2016 18:39:33 GMT
Yeah, it reflects the last offset in each partition all the time. The one
thing to be careful of is that it can be emitted from any replica
(whichever is the leader). So our tools that use the information always
look at all brokers and then take the larger non-null value.

We've been using it for a while now, and it seems pretty safe. At least
none of our Kafka developers have complained about us doing it :)

-Todd

On Wednesday, July 6, 2016, Kristoffer Sjögren <stoffe@gmail.com> wrote:

> That's awesome! I can see the JMX bean [1] in our current 0.8.2
> brokers and the number seems updated in real time per partition? I
> cannot find any official documentation on it? How safe is it to use?
>
> Also we're planning on upgrading to 0.10.0.0 - is it still available?
>
> Cheers,
> -Kristoffer
>
> [1] kafka.log:type=Log,name=LogEndOffset,topic=xxx,partition=xxx
>
> On Wed, Jul 6, 2016 at 5:36 PM, Todd Palino <tpalino@gmail.com
> <javascript:;>> wrote:
> > We do this through our monitoring agents by pulling it as a metric from
> the
> > LogEndOffset beans. By putting it into our metrics system we get a
> mapping
> > of timestamp to offset for every partition with (currently) 60 second
> > granularity. Useful for offset resets and other tasks.
> >
> > -Todd
> >
> > On Wednesday, July 6, 2016, Kristoffer Sjögren <stoffe@gmail.com
> <javascript:;>> wrote:
> >
> >> Hi
> >>
> >> Is there a way to get the last offset written by all partitions of a
> >> topic programmatically using the 0.10.0.0 API?
> >>
> >> At the moment I use KafkaConsumer.seekToEnd as seen in this gist[1]
> >> but maybe there is a better, more efficient, way to do it?
> >>
> >> Cheers,
> >> -Kristoffer
> >>
> >> [1] https://gist.github.com/krisskross/a49e462bedb89505e372672cd81129ab
> >>
> >
> >
> > --
> > *Todd Palino*
> > Staff Site Reliability Engineer
> > Data Infrastructure Streaming
> >
> >
> >
> > linkedin.com/in/toddpalino
>


-- 
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming



linkedin.com/in/toddpalino

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message