lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Collins <danwcoll...@gmail.com>
Subject Re: Limit the documents for each shard in solr cloud
Date Thu, 07 May 2015 07:11:58 GMT
Not sure I understand your problem.  If you have 20m documents, and 8
shards, then each shard is (broadly speaking) only going to have 2.5m docs
each, so I don't follow the 5m limit? That is with the default
routing/hashing, obviously you can write your own hash algorithm or you can
shard at your application level.

In terms of limiting documents in a shard, I'm not sure what purpose that
would serve.  If for arguments sake you only had 2 shards, and a limit of
5m doccs per shard, what happens when you hit that limit?  If you have
indexed 10m docs, and now you try to index one more, what would you expect
to happen, would the system just reject any documents, should it try to
shard to shard 1 but see that is full, and then fail-over to shard2 instead
(that's not going to work as sharding needs to be reproducible and the
document was intended for shard 1)?

Solr's basic premise would be to index what you gave it, as long as it has
sufficient space to do that.  If you want to limit your index to 20m docs,
that is probably better done at the application layer (but I still don't
really see why you would want to do that).

On 7 May 2015 at 06:29, Jilani Shaik <jilani24239@gmail.com> wrote:

> Hi,
>
> Is it possible to restrict number of documents per shard in Solr cloud?
>
> Lets say we have Solr cloud with 4 nodes, and on each node we have one
> leader and one replica. Like wise total we have 8 shards that includes
> replicas. Now I need to index my documents in such a way that each shard
> will have only 5 million documents. Total documents in Solr cloud should be
> 20 million documents.
>
>
> Thanks,
> Jilani
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message