kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kristoffer Sjögren <sto...@gmail.com>
Subject Re: Last offset in all partitions
Date Wed, 06 Jul 2016 18:18:11 GMT
That's awesome! I can see the JMX bean [1] in our current 0.8.2
brokers and the number seems updated in real time per partition? I
cannot find any official documentation on it? How safe is it to use?

Also we're planning on upgrading to 0.10.0.0 - is it still available?

Cheers,
-Kristoffer

[1] kafka.log:type=Log,name=LogEndOffset,topic=xxx,partition=xxx

On Wed, Jul 6, 2016 at 5:36 PM, Todd Palino <tpalino@gmail.com> wrote:
> We do this through our monitoring agents by pulling it as a metric from the
> LogEndOffset beans. By putting it into our metrics system we get a mapping
> of timestamp to offset for every partition with (currently) 60 second
> granularity. Useful for offset resets and other tasks.
>
> -Todd
>
> On Wednesday, July 6, 2016, Kristoffer Sjögren <stoffe@gmail.com> wrote:
>
>> Hi
>>
>> Is there a way to get the last offset written by all partitions of a
>> topic programmatically using the 0.10.0.0 API?
>>
>> At the moment I use KafkaConsumer.seekToEnd as seen in this gist[1]
>> but maybe there is a better, more efficient, way to do it?
>>
>> Cheers,
>> -Kristoffer
>>
>> [1] https://gist.github.com/krisskross/a49e462bedb89505e372672cd81129ab
>>
>
>
> --
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino

Mime
View raw message