kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu, Libo " <libo...@citi.com>
Subject RE: Is it possible to get latest offset from kafka server?
Date Mon, 01 Jul 2013 13:47:47 GMT
Thanks for your comment, Jun. Your last sentence is really helpful.



-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com] 
Sent: Sunday, June 30, 2013 11:52 PM
To: users@kafka.apache.org
Subject: Re: Is it possible to get latest offset from kafka server?

If you use MirrorMaker to mirror the data from the primary cluster to the backup cluster,
currently the offsets are not preserved btw the two clusters. In other words, the offsets
for the same message in the two clusters are not necessarily the same.



On Sat, Jun 29, 2013 at 10:56 AM, Yu, Libo <libo.yu@citi.com> wrote:

> Hi,
> I know kafka.tools.ConsumerOffsetChecker can find out the latest 
> offset for a topic from log files internally. Is there any public API 
> that users can use for that?
> This would be useful for the failover scenario. When one cluster is 
> down, the publisher will publish to the backup cluster. But the backup 
> cluster always lags behind. If it is possible for publisher to find 
> out the latest offset from the backup, it can always publish from that 
> offset assuming publisher has a large enough buffer.  This guarantees 
> no message loss. Thanks.
> Libo

View raw message