lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject RE: How large is your solr index?
Date Wed, 07 Jan 2015 20:50:54 GMT
Joseph Obernberger [joeo@lovehorsepower.com] wrote:

[HDFS, 9M docs, 2.9TB, 22 shards, 11 bare metal boxes]

> A typical query takes about 7 seconds to run, but we also do faceting
> and clustering.  Those can take in the 3 - 5 minute range depends on
> what was queried, but can be as little as 10 seconds. The index contains
> about 100 fields.

7 seconds without faceting seems like a long time. I am guessing your 3M daily updates are
spread throughout the day, instead of being a nightly batch job? How many concurrent searches
are you handling?

We have no experience with HDFS for Solr indexes, but a quick check indicates that it is not
a good fit for Solr. At least not out of the box: http://hbase.apache.org/book.html#perf.hdfs.curr

We did at one point try to use networked storage for our index. That meant 1/3 performance,
compared to local storage, but of course your mileage will vary. As you are looking into ways
of improving performance, what about testing the performance difference with local storage
(SSD of course)?

- Toke Eskildsen

Mime
View raw message