lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen>
Subject Re: Solr4 distributed IDF
Date Mon, 03 Sep 2012 08:09:53 GMT
On Fri, 2012-08-31 at 02:25 +0200, Lance Norskog wrote:
> The math for "confidence values" in probability theory shows that
> distributed DF does not matter after not very many documents. If you
> have 10s of thousands of documents in each shard, don't worry.

The old advice of distributing the documents by hashing id or a similar
deterministic method is sound enough. However, it is my experience that
sharding is often done by source or material: When building a workflow,
it is the logical thing to do. This might be more of an educational than
a technical problem.

For setups with a large unchanging set of data and a smaller set with
high update frequency, the standard advice is to have a large unchanging
shard and a smaller NRT one. For that case, I would expect that the
unchanging data is often quite different from the changing ones.

Third case: Distributed search where the separate indexes are controlled
by different parties, where the parties does want to collaborate on the
distribution part but does not want to have their data indexed by the
other parties. We currently have this challenge.

Toke Eskildsen

View raw message