lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Steele <ryan.ste...@pgi.com>
Subject Re: SolrCloud 5.0 cluster RAM requirements
Date Thu, 02 Apr 2015 22:46:34 GMT
Thank you Shawn and Toke for the information and links!

No, I was not the one on #solr IRC channel. :/

Here are the details I have right now:

I'm building/running the operations side of this new SolrCloud cluster. 
It will be in Amazon, the initial cluster I'm planning to start with is 
5 r3.xlarge instances each using a general purpose SSD EBS volume for 
the SolrCloud related data (this will be separate from the EBS volume 
used by the OS). Each instance has 30.5 GiB RAM--152.5 GiB cluster 
wide--and each instance has 4 vCPU's. I'm using Oracle Java 1.8.0_31 and 
the G1 GC.

The data will be indexed on a separate machine and added to the 
SolrCloud cluster while searching is taking place. Unfortunately I don't 
have numbers at this time on how much data will be indexed. I do know 
that we will have over 2000 collections--some will be small (a few 
hundred documents and only a few megabytes at most), and a few will be 
very large (somewhere in the gigabytes). Our old Solr Master/Slave 
systems isn't broken up this way, so we aren't certain about how exactly 
things will map out in SolrCloud.

I'll continue researching, but I expect I'll just have to monitor the 
cluster as data gets imported into it and make adjustments as needed.

Ryan

On 4/2/15 12:06 AM, Toke Eskildsen wrote:
> Ryan Steele<ryan.steele@pgi.com>  wrote:
>> Does a SolrCloud 5.0 cluster need enough RAM across the cluster to load
>> all the collections into RAM at all times?
> Although Shawn is right about us not being able to answer properly, sometimes we can
give qualified suggestions and guesses. At least to the direction you should be looking. The
quality of the guesses goes up with the amount of information provided and "1TB" is really
not much information.
>
> - Are you indexing while searching? How much?
> - How many documents in the index?
> - What is a typical query? What about faceting?
> - How many concurrent queries?
> - Expected median response time?
>
>> I'm building a SolrCloud cluster that may have approximately 1 TB of
>> data spread across the collections.
> We're running a 22TB SolrCloud of a single 16-core server with 256GB RAM. We've also
had performance problems serving a 100GB index from a same-size machine.
>
> The one hardware advice I will give is to start with SSDs and scale from there. With
present day price/performance, using spinning drives for anything IO-intensive makes little
sense.
>
> - Toke Eskildsen
>
>
---------------------------------------------------------------------------------------
 This email has been scanned for email related threats and delivered safely by Mimecast.
 For more information please visit http://www.mimecast.com
---------------------------------------------------------------------------------------


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message