lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay Kokatnur <kokatnur.vi...@gmail.com>
Subject Re: DIH Speed
Date Fri, 28 Apr 2017 03:53:39 GMT
‚ÄčLet me clarify -

DIH is running on Solr 6.5.0 that calls a different solr instance running‚Äč
on 4.5.0, which has 150M documents.  If we try fetch them using DIH onto
new solr cluster, wouldn't it result in deep paging on solr 4.5.0 and
drastically slow down indexing on solr 6.5.0?

On Thu, Apr 27, 2017 at 4:40 PM, Erick Erickson <erickerickson@gmail.com>
wrote:

> I'm unclear why DIH an deep paging are  mixed. DIH is
> indexing and deep paging is querying.
>
> If it's querying, consider cursorMark or the /export handler.
> https://lucidworks.com/2013/12/12/coming-soon-to-solr-
> efficient-cursor-based-iteration-of-large-result-sets/
>
> If it's DIH, please explain a bit more.
>
> Best,
> Erick
>
> On Thu, Apr 27, 2017 at 3:37 PM, Vijay Kokatnur
> <kokatnur.vijay@gmail.com> wrote:
> > We have a new solr 6.5.0 cluster, for which data is being imported via
> DIH
> > from another Solr cluster running version 4.5.0.
> >
> > This question comes back to deep paging, but we have observed that after
> 30
> > minutes of querying the rate of processing goes down from 400/s to about
> > 120/s.  At that point it has processed only 500K of 1.3M docs.  Is there
> > any way to speed this up?
> >
> > And, I can't go back to the source for the data.
> >
> > --
>



-- 
Best,
Vijay

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message