lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geert-Jan Brits <gbr...@gmail.com>
Subject Re: How do i update some document when i use sharding indexs?
Date Mon, 09 Aug 2010 10:06:09 GMT
Just to be completely clear: the program that splits your index in 20 shards
should employ this algo as well.


2010/8/9 Geert-Jan Brits <gbrits@gmail.com>

> I'm not sure if Solr has some build-in support for sharding-functions, but
> you should generally use some hashing-algorithm to split the indices and use
> the same hash-algorithm to locate which shard contains a document.
> http://en.wikipedia.org/wiki/Hash_function
>
> Without employing any domain knowledge (of documents you possible want to
> group toegether on a single shard for performance) you could build a very
> simple (crude) hash-function by md5-hashing the unique-keys of your
> documents, taking the first 3 chars (should be precise enough, so load is
> pretty much balanced), calculate a nr from the chars (256 * first char + 16
> * 2nd char + 3rd char), and take that nr modulo 20. That should give you a
> nr in [0,20) which is the shard-index.
>
> use the same algorithm to determine which shard contains the document that
> you want to change.
>
> Geert-Jan
>
>
> 2010/8/9 lu.rongbin <lu.rongbin@goodhope.net>
>
>
>>    My index has 76 million documents, I split it to 20 indexs because the
>> size of index is 33G. I deploy 20 shards for search response performence
>> on
>> ec2's 20 instances.But when i wan't to update some doc, it means i must
>> traversal each index , and find the document is in which shard index, and
>> update the doc? It's crazy! How can i do?
>>    thanks.
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/How-do-i-update-some-document-when-i-use-sharding-indexs-tp1053509p1053509.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message