From dev-return-189972-apmail-lucene-dev-archive=lucene.apache.org@lucene.apache.org Wed Dec 24 19:36:14 2014 Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D74FC10A15 for ; Wed, 24 Dec 2014 19:36:14 +0000 (UTC) Received: (qmail 52132 invoked by uid 500); 24 Dec 2014 19:36:14 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 52060 invoked by uid 500); 24 Dec 2014 19:36:14 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 52048 invoked by uid 99); 24 Dec 2014 19:36:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Dec 2014 19:36:13 +0000 Date: Wed, 24 Dec 2014 19:36:13 +0000 (UTC) From: "Yonik Seeley (JIRA)" To: dev@lucene.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SOLR-6810?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14258= 456#comment-14258456 ]=20 Yonik Seeley edited comment on SOLR-6810 at 12/24/14 7:35 PM: -------------------------------------------------------------- Thanks for pulling out the algorithm Shalin. I'm still not quite groking i= t though... do you understand it well enough to give a high level descripti= on for those who know Solr but who haven't looked at the patch? As in... w= hat's the high level description of what this patch implements? The main idea seems to be: you don't need IDs to merge the top docs from ea= ch shard. Correct? Are there other ideas/optimizations in this patch? edit: Also, does this patch also improve things if docValues are used for t= he ID field? was (Author: yseeley@gmail.com): Thanks for pulling out the algorithm Shalin. I'm still not quite groking i= t though... do you understand it well enough to give a high level descripti= on for those who know Solr but who haven't looked at the patch? As in... w= hat's the high level description of what this patch implements? The main idea seems to be: you don't need IDs to merge the top docs from ea= ch shard. Correct? Are there other ideas/optimizations in this patch? > Faster searching limited but high rows across many shards all with many h= its > -------------------------------------------------------------------------= --- > > Key: SOLR-6810 > URL: https://issues.apache.org/jira/browse/SOLR-6810 > Project: Solr > Issue Type: Improvement > Components: search > Reporter: Per Steffensen > Assignee: Shalin Shekhar Mangar > Labels: distributed_search, performance > Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.pat= ch, branch_5x_rev1645549.patch > > > Searching "limited but high rows across many shards all with many hits" i= s slow > E.g. > * Query from outside client: q=3Dsomething&rows=3D1000 > * Resulting in sub-requests to each shard something a-la this > ** 1) q=3Dsomething&rows=3D1000&fl=3Did,score > ** 2) Request the full documents with ids in the global-top-1000 found am= ong the top-1000 from each shard > What does the subject mean > * "limited but high rows" means 1000 in the example above > * "many shards" means 200-1000 in our case > * "all with many hits" means that each of the shards have a significant n= umber of hits on the query > The problem grows on all three factors above > Doing such a query on our system takes between 5 min to 1 hour - dependin= g on a lot of things. It ought to be much faster, so lets make it. > Profiling show that the problem is that it takes lots of time to access t= he store to get id=E2=80=99s for (up to) 1000 docs (value of rows parameter= ) per shard. Having 1000 shards its up to 1 mio ids that has to be fetched.= There is really no good reason to ever read information from store for mor= e than the overall top-1000 documents, that has to be returned to the clien= t. > For further detail see mail-thread "Slow searching limited but high rows = across many shards all with high hits" started 13/11-2014 on dev@lucene.apa= che.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org