lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: LotsOfCores feature
Date Thu, 06 Jun 2013 19:53:59 GMT
100K is really not the limit, it's just hard to imagine
100K cores on a single machine unless some were
really rarely used. And it's per node, not cluster-wide.

The current state is that everything is in place, including
transient cores, auto-discovery, etc. So you should be
able to go ahead and try it out.

The next bit that will help with efficiency is sharing named
config sets. The intent here is that <solrhome>/configs will
contain sub-dirs like "conf1", "conf2" etc. Then your cores
can reference configName=conf1 and only one copy of
the configuration data will be used rather than re-loading one
for each core as it comes up and down.

Do note that the _first_ query in to one of the not-yet-loaded
cores will be slow. The model here is that you can tolerate
some queries taking more time at first than you might like
in exchange for the hardware savings. This pre-supposes that
you simply cannot fit all the cores into memory at once.

The "won't fix" bits are there because, as we got farther into this
process, the approach changed and the functionality of the
won't fix JIRAs was subsumed by other changes by and large.

I've got to update that documentation sometime, but just haven't
had time yet. If you go down this route, we'll be happy to
add your name to the authorized editors of the wiki list if you'd
like.

Best
Erick

On Thu, Jun 6, 2013 at 3:08 PM, Aleksey <bittercold@gmail.com> wrote:
> I was looking at this wiki and linked issues:
> http://wiki.apache.org/solr/LotsOfCores
>
> they talk about a limit being 100K cores. Is that per server or per
> entire fleet because zookeeper needs to manage that?
>
> I was considering a use case where I have tens of millions of indices
> but less that a million needs to be active at any time, so they need
> to be loaded on demand and evicted when not used for a while.
> Also since number one requirement is efficient loading of course I
> assume I will store a prebuilt index somewhere so Solr will just
> download it and strap it in, right?
>
> The root issue is marked as "won;t fix" but some other important
> subissues are marked as resolved. What's the overall status of the
> effort?
>
> Thank you in advance,
>
> Aleksey

Mime
View raw message