lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: How large is your solr index?
Date Thu, 08 Jan 2015 15:08:53 GMT
On 1/8/2015 4:37 AM, Bram Van Dam wrote:
> Hmm. That is a good point. I wonder if there's some kind of middle
> ground here? Something that lets me send an update (or new document) to
> an arbitrary node/shard but which is still routed according to my
> specific requirements? Maybe this can already be achieved by messing
> with the routing?

<snip>

> That's fine. We have a lot of query (pre-)processing outside of Solr.
> It's no problem for us to send a couple of queries to a couple of shards
> and aggregate the result ourselves. It would, of course, be nice if
> everything worked in distributed mode, but at least for us it's not an
> issue. This is a side effect of our complex reporting requirements -- we
> do aggregation, filtering and other magic on data that is partially in
> Solr and partially elsewhere.

SolrCloud, when you do fully automatic document routing, does handle
everything for you.  You can query any node and send updates to any
node, and they will end up in the right place.  There is currently a
strong caveat: Indexing performance sucks when updates are initially
sent to the wrong node.  The performance hit is far larger than we
expected it to be, so there is an issue in Jira to try and make that
better.  No visible work has been done on the issue yet:

https://issues.apache.org/jira/browse/SOLR-6717

The Java client (SolrJ, specifically CloudSolrServer) sends all updates
to the correct nodes, because it can access the clusterstate and knows
where updates need to go and where the shard leaders are.

> This is a very good point. But I don't think SPLITSHARD is the magical
> answer here. If you have N shards on N boxes, and they are all getting
> nearly "full" and you decide to split one and move half to a new box,
> you'll end up with N-2 nearly full boxes and 2 half-full boxes. What
> happens if the disks fill up further? Do I have to split each shard?
> That sounds pretty nightmareish!

Planning ahead for growth is critical with SolrCloud, but there is
something you can do if you discover that you need to radically
re-shard:  Create a whole new collection with the number of shards you
want, likely using the original set of Solr servers plus some new ones.
 Rebuild the index into that collection.  Delete the old collection, and
create a collection alias pointing the original name at the new
collection.  The alias will work for both queries and updates.

Thanks,
Shawn


Mime
View raw message