kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Is it possible to get latest offset from kafka server?
Date Mon, 01 Jul 2013 03:51:59 GMT
If you use MirrorMaker to mirror the data from the primary cluster to the
backup cluster, currently the offsets are not preserved btw the two
clusters. In other words, the offsets for the same message in the two
clusters are not necessarily the same.



On Sat, Jun 29, 2013 at 10:56 AM, Yu, Libo <libo.yu@citi.com> wrote:

> Hi,
> I know kafka.tools.ConsumerOffsetChecker can find out the latest offset for
> a topic from log files internally. Is there any public API that users can
> use for that?
> This would be useful for the failover scenario. When one cluster is down,
> the
> publisher will publish to the backup cluster. But the backup cluster
> always lags
> behind. If it is possible for publisher to find out the latest offset from
> the backup,
> it can always publish from that offset assuming publisher has a large
> enough
> buffer.  This guarantees no message loss. Thanks.
> Libo

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message