lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Poor performance on distributed search
Date Mon, 19 Dec 2011 03:01:33 GMT
Here's another test. Add &debugQuery=on to your query and post the results.
I believe you'll find that the QTime parameter returned in the packet
is quite small. That is the amount of time spent in the query, NOT
the time spent reading the docs form disk to return. And if that number
remains small whether you search 10 or 2,000 docs, then your
problem is just that you're reading 2,000 documents from disk
and returning them. And that has nothing to do with sharding, it's
just how things work....

Although there's one other possibility: By returning 2,000 rows, you
require that each shard assemble a list of the top 2,000 documents
and then they are collated into a single packet, so you're asking
the system to do a lot of list processing.

Also, I really have a hard time seeing what advantage you get from
putting all those shards on the same machine, you're just creating
extra work.

Best
Erick

On Sat, Dec 17, 2011 at 4:11 PM, ku3ia <demesg@gmail.com> wrote:
> Hi, Erick!
>
>>>Right, are you falling afoul of the recursive shard thing? That is,
>>>if you shards point back to itself. As far as I understand, your
>>>shards parameter in your request handler shouldn't point back
>>>to itself....
> No, my request handler don't points itself cause default is false. But I
> tested today, using classic shards parameter. Here is results:
>
> successfully started at 22:25:58
> Queries interval is: 10 queries per minute
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(jurisdictions)&rows=2000&start=0&fl=*,score&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> ...
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(mainstreaming)&rows=2000&start=0&fl=*,score&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> utility successfully stopped at 22:30:58
> Elapsed time: 300 secs
> --- solr ---
> Queries processed: 11
> Queries cancelled: 39
> Average QTime is: 55966.8 ms
> Average RTime is: 56.2727 sec(s)
> Size of data-dir is: 496726 bytes
>
> And test with RecordID only:
> successfully started at 22:17:16
> Queries interval is: 10 queries per minute
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(simulating)&rows=2000&start=0&fl=RecordID&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> ...
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(multimillionaire's)&rows=2000&start=0&fl=RecordID&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> utility successfully stopped at 22:22:16
> Elapsed time: 300 secs
> --- solr ---
> Queries processed: 23
> Queries cancelled: 27
> Average QTime is: 13991.8 ms
> Average RTime is: 14.1739 sec(s)
> Size of data-dir is: 223580 bytes
>
> And read speed is the same.
>
> On production I create a RAMDisk (mount -t tmpfs -o size=xxxM tmpfs
> /mnt/tmpfs/) and put my shards onto it. The QTime/RTime was magnificent. But
> the problem is, I dont have much RAM for it(
>
> Can it be my huge index, terms or maybe it's to much queries per minute, or
> row count or I made something wrong in my configs? And, can you please watch
> drive speed on your indexes? Is it the same?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3594683.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message