lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject RE: Solr performance issues
Date Mon, 29 Dec 2014 22:24:48 GMT
Mahmoud Almokadem [prog.mahmoud@gmail.com] wrote:
> I've the same index with a bit different schema and 200M documents,
> installed on 3 r3.xlarge (30GB RAM, and 600 General Purpose SSD). The size
> of index is about 1.5TB, have many updates every 5 minutes, complex queries
> and faceting with response time of 100ms that is acceptable for us.

So you have
Setup 1: 3 * (30GB RAM + 600GB SSD) for a total of 1.5TB index 200M docs. Acceptable performance.
Setup 2: 3 * (60GB RAM + 1TB SSD + 500GB SSD) for a total of 3.3TB 350M docs. Poor performance.

The only real difference, besides doubling everything, is the LVM? I understand why you find
that to be the culprit, but from what I can read, the overhead should not be anywhere near
enough to result in the performance drop you are describing. Could it be that some snapshotting
or backup was running when you tested?

Splitting your shards and doubling the number of machines, as you suggest, would result in
Setup 3: 6 * (60GB RAM + 600GB SSD) for a total of 3.3TB 350M docs.
which would be remarkable similar to your setup 1. I think that would be the next logical
step, unless you can easily do a temporary boost of your IOPS.

BTW: You are getting dangerously close to your storage limits here - it seems that a single
large merge could make you run out of space.

- Toke Eskildsen

Mime
View raw message