lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Storing external transaction log-ids in lucene...
Date Thu, 10 Aug 2017 12:39:22 GMT
IW.setCommitData (now .setLiveCommitData in 7.0) is the right way to store
this.

Mike McCandless

http://blog.mikemccandless.com

On Thu, Aug 10, 2017 at 6:57 AM, Ravikumar Govindarajan <
ravikumar.govindarajan@gmail.com> wrote:

> Every mutation (Add/Update/Delete) has a transaction-id (incremental long)
> assigned by our Messaging Queue (Kafka)
>
> To index these mutations, an indexer thread pulls data from the queue, adds
> & commits to IndexWriter, then updates the latest transaction-id in an
> external system (ZooKeeper). During rollback/server-restarts, the threads
> read the previous value from ZK & resume...
>
> Have now moved to NRT implementation, where commit thread runs in the
> background & am unable to find the latest transaction-id that got committed
>
> Initially thought of adding transaction-id as a first-class field of the
> index itself & then writing a custom-codec to harvest it, but this fails
> when a mutation-set consists only of deletes [as there is no way to pass
> these ids via IW delete APIs]
>
> Is there a way to store this as part of segment-info?. Can setCommitData()
> of IW help me here in any way?
>
> --
> Ravi
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message