lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gregg Donovan <gregg...@gmail.com>
Subject Solr 4.0 SnapPuller version vs. generation issue
Date Thu, 10 Jan 2013 21:11:27 GMT
We are in the midst of upgrading from Solr 3.6 to Solr 4.0 and have
encountered an issue with the method the SnapPuller now uses to determine
if a new directory is needed when fetching files to a slave from master.

With Solr 3.6, our reindexing process was:

On master:
1. Reindex in a separate process into a new directory:
<solr.data.dir>/<core>/index-<timestamp_of_reindex_start>
2. Update <solr.data.dir>/<core>/index.properties to 'index=<solr.data.dir>
/<core>/index-<timestamp_of_reindex_start>'
3. Reload <core> on master so that the new index referenced in
<solr.data.dir>/<core>/index.properties would be loaded.

Slaves would then fetch this new index into a new directory without any
manual intervention because the slave would determine that a full new index
copy was needed. SnapPuller in 3.6 used:

*boolean* isFullCopyNeeded = commit.getGeneration() >= latestGeneration;

Since the generation on master would be near zero on master after a reindex
and larger on a slave, the new index would be placed in a new directory on
the slave.

In Solr 4.0, beginning with svn 1235888 [1], the check is now:

*boolean* isFullCopyNeeded =
IndexDeletionPolicyWrapper.*getCommitTimestamp*(commit)
>= latestVersion || forceReplication;
As far as I can tell, forceReplication is only used in SolrCloud recovery
scenarios. Our new index on master has a newer  commitTimeMSec than the
slave index, so neither of these conditions is true. With
isFullCopyNeeded=false, our new index files get pulled into the existing
directory on the slave where they are then deleted. I think this is because
their generation is older than the current slave generation.

Would it still make sense to create a new directory on the slave if
master's generation is less than the slave's generation? I can't see a
scenario where you'd want a slave to fetch files from master with a smaller
generation into the current index directory of a slave.

If the commitTimeMSec based check in Solr 4.0 is needed for SolrCloud, what
other methods of swapping in an entirely new index would you recommend for
those not using SolrCloud?

[1] --
http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.java?r1=1144761&r2=1235888&pathrev=1235888&diff_format=h

Thanks!

--Gregg


Gregg Donovan
Senior Software Engineer, Etsy.com
gregg@etsy.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message