kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Offset retrieval
Date Mon, 03 Oct 2011 20:03:31 GMT
You can do it by messing directly with zookeeper, but I think we need an API
that does this in a proper way. I think commit() should return the offset
committed and there should also be some kind of revert(offsets) which
restores to a previous state.

-Jay

On Mon, Oct 3, 2011 at 12:22 PM, Mathias Herberts <
mathias.herberts@gmail.com> wrote:

> In some use cases it would be useful to be able to retrieve the last
> committed offsets of a connector and also be able to force those
> offsets at a later time (for example when restarting an app).
>
> Typical usecase is index creation. Imagine you have a Lucene index
> which you are building. At (index) commit time, you add commit
> metadata which includes the current offsets. Should the app crash, you
> can retrieve the metadata from the last index commit and could start
> consuming messages right where you last stopped (i.e. at the time of
> last index commit).
>
> Is there any way to do that at the moment? Maybe fiddling with znodes
> directly?
>
> Mathias.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message