lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Uneven index distribution using composite router
Date Thu, 26 Mar 2015 17:37:28 GMT
right, when you take over routing, making sure the distribution is
even is now your responsibility.

Your assumption is that the amount of _text_ in each doc is roughly
the same between your three languages, have you verified this? And are
you doing anything like copyFields that are kicking in on one shard
but not the others (e.g. if you have text_en fields you might be
copying that to text_en_all but not doing so with text_ger to
text_ger_all). that's totally a shot in the dark though.

Best,
Erick

On Thu, Mar 26, 2015 at 10:26 AM, Shamik Bandopadhyay <shamikb@gmail.com> wrote:
> Hi,
>
>    I'm using a three level composite router in a solr cloud environment,
> primarily for multi-tenant and field collapsing. The format is as follows.
>
> *language!topic!url*.
>
> An example would be :
>
> ENU!12345!www.testurl.com/enu/doc1
> GER!12345!www.testurl.com/ger/doc2
> CHS!67890!www.testurl.com/chs/doc3
>
> The Solr Cloud cluster contains 2 shard, each having 3 replicas. After
> indexing around 10 million documents, I'm observing that the index size in
> shard 1 is around 60gb while shard 2 is 15gb. So the bulk of the data is
> getting indexed in shard 1. Since 60% of the document is english, I expect
> the index size to be higher on one shard, but the difference seem little
> too high.
>
> The idea is to make sure that all ENU!12345 documents are routed to one
> shard so that distributed field collapsing works. Is there something I can
> do differently here to make a better distribution ?
>
> Any pointers will be appreciated.
>
> Regards,
> Shamik

Mime
View raw message