lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noble Paul നോബിള്‍ नोब्ळ् <noble.p...@gmail.com>
Subject Re: LotsOfCores feature
Date Mon, 10 Jun 2013 09:27:20 GMT
Aleksey,

It was a less than ideal situation. because we did not have a choice. We
had external systems/scripts to manage this. A new custom implementation is
being built on SolrCloud which would have taken care of most of hose
 issues.

SolrReplication is a hidden once you move to cloud. But it will continue in
the same way if you have a stand-lone deployment.


On Mon, Jun 10, 2013 at 1:20 AM, Aleksey <bittercold@gmail.com> wrote:

> Thanks Paul. Just a little clarification:
>
> You mention that you migrate data using built-in replication, but if
> you map and route users yourself, doesn't that mean that you also need
> to manage replication yourself? Your routing logic needs to be aware
> of how to map both replicas for each user, and if one hosts goes down,
> then it needs to distribute traffic that it was receiving over other
> hosts. Same thing for adding more hosts.
> I did a couple of quick searches and found mostly older wikis that say
> solr replication will change in the future. Would you be able to point
> me to the right one?
>
>
> -
>
> On Fri, Jun 7, 2013 at 8:34 PM, Noble Paul നോബിള്‍  नोब्ळ्
> <noble.paul@gmail.com> wrote:
> > We set it up like this
> > + individual solr instances are setup
> > + external mapping/routing to allocate users to instances. This
> information
> > can be stored in an external data store
> > + all cores are created as transient and loadonstart as false
> > + cores come online on demand
> > + as and when users data get bigger (or hosts are hot)they are migrated
> > between less hit hosts using in built replication
> >
> > Keep in mind we had the schema for all users. Currently there is no way
> to
> > upload a new schema to solr.
> > On Jun 8, 2013 1:15 AM, "Aleksey" <bittercold@gmail.com> wrote:
> >
> >> > Aleksey: What would you say is the average core size for your use
> case -
> >> > thousands or millions of rows? And how sharded would each of your
> >> > collections be, if at all?
> >>
> >> Average core/collection size wouldn't even be thousands, hundreds more
> >> like. And the largest would be half a million or so but that's a
> >> pathological case. I don't need sharding and queries than fan out to
> >> different machines. If fact I'd like to avoid that so I don't have to
> >> collate the results.
> >>
> >>
> >> > The Wiki page was built not for Cloud Solr.
> >> >
> >> > We have done such a deployment where less than a tenth of cores were
> >> active
> >> > at any given point in time. though there were tens of million indices
> >> they
> >> > were split among a large no:of hosts.
> >> >
> >> > If you don't insist of Cloud deployment it is possible. I'm not sure
> if
> >> it
> >> > is possible with cloud
> >>
> >> By Cloud you mean specifically SolrCloud? I don't have to have it if I
> >> can do without it. Bottom line is I want a bunch of small cores to be
> >> distributed over a fleet, each core completely fitting on one server.
> >> Would you be willing to provide a little more details on your setup?
> >> In particular, how are you managing the cores?
> >> How do you route requests to proper server?
> >> If you scale the fleet up and down, does reshuffling of the cores
> >> happen automatically or is it an involved manual process?
> >>
> >> Thanks,
> >>
> >> Aleksey
> >>
>



-- 
-----------------------------------------------------
Noble Paul

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message