lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] [Commented] (LUCENE-5438) add near-real-time replication
Date Tue, 09 Feb 2016 10:07:18 GMT


Michael McCandless commented on LUCENE-5438:

I got back to this issue and have been pushing changes here:;a=shortlog;h=refs/heads/jira/lucene-5438-nrt-replication

Net/net I think it's in pretty good shape ... I'd like to add this for
6.0 as an experimental feature, as an alternative to document based

I think there are complex tradeoffs between the two approaches to
distributing Lucene, but I think it's important users at least have
a choice.

With this change, multiple nodes (primary and replicas) have
essentially the same transactional semantics that a single Lucene
IndexWriter + NRT readers offers.  You have known point-in-time views
that are the consistent (long version) across nodes, you can commit
any node (primary or replica), rollback etc.  When things crash, on
restart they are back to the last commit, etc.

The test cases are quite evil: they spawn sub-process JVMs, and
communicate over a naive (thread-per-connection) TCP protocol to copy
files, index documents, search, etc.  And they randomly crash (thank
you {{TestIndexWriterOnJRECrash}}!), commit, close, flip bits during
file copy, simulate slow networks, etc.

I'll make an applyable patch from the current branch and post here.

> add near-real-time replication
> ------------------------------
>                 Key: LUCENE-5438
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: modules/replicator
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 6.0
>         Attachments: LUCENE-5438.patch, LUCENE-5438.patch, LUCENE-5438.patch
> Lucene's replication module makes it easy to incrementally sync index
> changes from a master index to any number of replicas, and it
> handles/abstracts all the underlying complexity of holding a
> time-expiring snapshot, finding which files need copying, syncing more
> than one index (e.g., taxo + index), etc.
> But today you must first commit on the master, and then again the
> replica's copied files are fsync'd, because the code operates on
> commit points.  But this isn't "technically" necessary, and it mixes
> up durability and fast turnaround time.
> Long ago we added near-real-time readers to Lucene, for the same
> reason: you shouldn't have to commit just to see the new index
> changes.
> I think we should do the same for replication: allow the new segments
> to be copied out to replica(s), and new NRT readers to be opened, to
> fully decouple committing from visibility.  This way apps can then
> separately choose when to replicate (for freshness), and when to
> commit (for durability).
> I think for some apps this could be a compelling alternative to the
> "re-index all documents on each shard" approach that Solr Cloud /
> ElasticSearch implement today, and it may also mean that the
> transaction log can remain external to / above the cluster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message