lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gregg Donovan <>
Subject Solr 4.0 SnapPuller version vs. generation issue
Date Thu, 10 Jan 2013 21:11:27 GMT
We are in the midst of upgrading from Solr 3.6 to Solr 4.0 and have
encountered an issue with the method the SnapPuller now uses to determine
if a new directory is needed when fetching files to a slave from master.

With Solr 3.6, our reindexing process was:

On master:
1. Reindex in a separate process into a new directory:
2. Update <>/<core>/ to 'index=<>
3. Reload <core> on master so that the new index referenced in
<>/<core>/ would be loaded.

Slaves would then fetch this new index into a new directory without any
manual intervention because the slave would determine that a full new index
copy was needed. SnapPuller in 3.6 used:

*boolean* isFullCopyNeeded = commit.getGeneration() >= latestGeneration;

Since the generation on master would be near zero on master after a reindex
and larger on a slave, the new index would be placed in a new directory on
the slave.

In Solr 4.0, beginning with svn 1235888 [1], the check is now:

*boolean* isFullCopyNeeded =
>= latestVersion || forceReplication;
As far as I can tell, forceReplication is only used in SolrCloud recovery
scenarios. Our new index on master has a newer  commitTimeMSec than the
slave index, so neither of these conditions is true. With
isFullCopyNeeded=false, our new index files get pulled into the existing
directory on the slave where they are then deleted. I think this is because
their generation is older than the current slave generation.

Would it still make sense to create a new directory on the slave if
master's generation is less than the slave's generation? I can't see a
scenario where you'd want a slave to fetch files from master with a smaller
generation into the current index directory of a slave.

If the commitTimeMSec based check in Solr 4.0 is needed for SolrCloud, what
other methods of swapping in an entirely new index would you recommend for
those not using SolrCloud?

[1] --



Gregg Donovan
Senior Software Engineer,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message