lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky" <j...@basetechnology.com>
Subject Re: LotsOfCores feature
Date Fri, 07 Jun 2013 13:59:18 GMT
AFAICT, SolrCloud addresses the use case of distributed update for a 
relatively smaller number of collections (dozens?) that have a relatively 
larger number of rows - billions over a modest to moderate number of nodes 
(a handful to a dozen or dozens). So, maybe dozens of collections (some 
people still call these "cores") that distribute hundreds of millions if not 
billions of rows over dozens (or potentially low hundreds) of nodes. 
Technically, ZK was designed for thousands of nodes, but I don't think that 
was for the use case of distributed query that constantly fans out to all 
shards.

Aleksey: What would you say is the average core size for your use case - 
thousands or millions of rows? And how sharded would each of your 
collections be, if at all?

-- Jack Krupansky

-----Original Message----- 
From: Noble Paul നോബിള്‍ नोब्ळ्
Sent: Friday, June 07, 2013 6:38 AM
To: solr-user@lucene.apache.org
Subject: Re: LotsOfCores feature

The Wiki page was built not for Cloud Solr.

We have done such a deployment where less than a tenth of cores were active
at any given point in time. though there were tens of million indices they
were split among a large no:of hosts.


If you don't insist of Cloud deployment it is possible. I'm not sure if it
is possible with cloud


On Fri, Jun 7, 2013 at 12:38 AM, Aleksey <bittercold@gmail.com> wrote:

> I was looking at this wiki and linked issues:
> http://wiki.apache.org/solr/LotsOfCores
>
> they talk about a limit being 100K cores. Is that per server or per
> entire fleet because zookeeper needs to manage that?
>
> I was considering a use case where I have tens of millions of indices
> but less that a million needs to be active at any time, so they need
> to be loaded on demand and evicted when not used for a while.
> Also since number one requirement is efficient loading of course I
> assume I will store a prebuilt index somewhere so Solr will just
> download it and strap it in, right?
>
> The root issue is marked as "won;t fix" but some other important
> subissues are marked as resolved. What's the overall status of the
> effort?
>
> Thank you in advance,
>
> Aleksey
>



-- 
-----------------------------------------------------
Noble Paul 


Mime
View raw message