lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar <susheel2...@gmail.com>
Subject Re: problems with bulk indexing with concurrent DIH
Date Tue, 02 Aug 2016 13:44:24 GMT
My experience with DIH was we couldn't scale to the level we wanted.  SorlJ
with multi-threading & batch updates (parallel threads pushing data into
solr) worked and were able to ingest 5K-10K docs per second.

Thanks,
Susheel

On Tue, Aug 2, 2016 at 9:15 AM, Mikhail Khludnev <mkhl@apache.org> wrote:

> Bernd,
> But why do you have so many deletes? Is it expected?
> When you run DIHs concurrently, do you shard intput data by uniqueKey?
>
> On Wed, Jul 27, 2016 at 6:20 PM, Bernd Fehling <
> bernd.fehling@uni-bielefeld.de> wrote:
>
> > If there is a problem in single index then it might also be in CloudSolr.
> > As far as I could figure out from INFOSTREAM, documents are added to
> > segments
> > and terms are "collected". Duplicate term are "deleted" (or whatever).
> > These deletes (or whatever) are not concurrent.
> > I have a lines like:
> > BD 0 [Wed Jul 27 13:28:48 GMT+01:00 2016; Thread-27879]: applyDeletes:
> > infos=...
> > BD 0 [Wed Jul 27 13:31:48 GMT+01:00 2016; Thread-27879]: applyDeletes
> took
> > 180028 msec
> > ...
> > BD 0 [Wed Jul 27 13:42:03 GMT+01:00 2016; Thread-27890]: applyDeletes:
> > infos=...
> > BD 0 [Wed Jul 27 14:38:55 GMT+01:00 2016; Thread-27890]: applyDeletes
> took
> > 3411845 msec
> >
> > 3411545 msec are about 56 minutes where the system is doing what???
> > At least not indexing because only one JAVA process and no I/O at all!
> >
> > How can SolrJ help me now with this problem?
> >
> > Best
> > Bernd
> >
> >
> > Am 27.07.2016 um 16:41 schrieb Erick Erickson:
> > > Well, at least it'll be easier to debug in my experience. Simple
> example.
> > > At some point you'll call CloudSolrClient.add(doc list). Comment just
> > that
> > > out and you'll be able to isolate whether the issue is querying the be
> or
> > > sending to Solr.
> > >
> > > Then CloudSolrClient (assuming SolrCloud) has efficiencies in terms of
> > > routing...
> > >
> > > Best
> > > Erick
> > >
> > > On Jul 27, 2016 7:24 AM, "Bernd Fehling" <
> bernd.fehling@uni-bielefeld.de
> > >
> > > wrote:
> > >
> > >> So writing some SolrJ doing the same job as the DIH script
> > >> and using that concurrent will solve my problem?
> > >> I'm not using Tika.
> > >>
> > >> I don't think that DIH is my problem, even if it is not the best
> > solution
> > >> right now.
> > >> Nevertheless, you are right SolrJ has higher performance, but what
> > >> if I have the same problems with SolrJ like with DIH?
> > >>
> > >> If it runs with DIH it should run with SolrJ with additional
> performance
> > >> boost.
> > >>
> > >> Bernd
> > >>
> > >>
> > >> On 27.07.2016 at 16:03, Erick Erickson:
> > >>> I'd actually recommend you move to a SolrJ solution
> > >>> or similar. Currently, you're putting a load on the Solr
> > >>> servers (especially if you're also using Tika) in addition
> > >>> to all indexing etc.
> > >>>
> > >>> Here's a sample:
> > >>> https://lucidworks.com/blog/2012/02/14/indexing-with-solrj/
> > >>>
> > >>> Dodging the question I know, but DIH sometimes isn't
> > >>> the best solution.
> > >>>
> > >>> Best,
> > >>> Erick
> > >>>
> > >>> On Wed, Jul 27, 2016 at 6:59 AM, Bernd Fehling
> > >>> <bernd.fehling@uni-bielefeld.de> wrote:
> > >>>> After enhancing the server with SSDs I'm trying to speed up
> indexing.
> > >>>>
> > >>>> The server has 16 CPUs and more than 100G RAM.
> > >>>> JAVA (1.8.0_92) has 24G.
> > >>>> SOLR is 4.10.4.
> > >>>> Plain XML data to load is 218G with about 96M records.
> > >>>> This will result in a single index of 299G.
> > >>>>
> > >>>> I tried with 4, 8, 12 and 16 concurrent DIHs.
> > >>>> 16 and 12 was to much because for 16 CPUs and my test continued
> with 8
> > >> concurrent DIHs.
> > >>>> Then i was trying different <indexConfig> and <updateHandler>
> settings
> > >> but now I'm stuck.
> > >>>> I can't figure out what is the best setting for bulk indexing.
> > >>>> What I see is that the indexing is "falling asleep" after some
time
> of
> > >> indexing.
> > >>>> It is only producing del-files, like _11_1.del, _w_2.del,
> _h_3.del,...
> > >>>>
> > >>>> <indexConfig>
> > >>>>     <maxIndexingThreads>8</maxIndexingThreads>
> > >>>>     <ramBufferSizeMB>1024</ramBufferSizeMB>
> > >>>>     <maxBufferedDocs>-1</maxBufferedDocs>
> > >>>>     <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> > >>>>       <int name="maxMergeAtOnce">8</int>
> > >>>>       <int name="segmentsPerTier">100</int>
> > >>>>       <int name="maxMergedSegmentMB">512</int>
> > >>>>     </mergePolicy>
> > >>>>     <mergeFactor>8</mergeFactor>
> > >>>>     <mergeScheduler
> > >> class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
> > >>>>     <lockType>${solr.lock.type:native}</lockType>
> > >>>>     ...
> > >>>> </indexConfig>
> > >>>>
> > >>>> <updateHandler class="solr.DirectUpdateHandler2">
> > >>>>      ### no autocommit at all
> > >>>>      <autoSoftCommit>
> > >>>>        <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> > >>>>      </autoSoftCommit>
> > >>>> </updateHandler>
> > >>>>
> > >>>>
> > >>>>
> > >>
> >
> command=full-import&optimize=false&clean=false&commit=false&waitSearcher=false
> > >>>> After indexing finishes there is a final optimize.
> > >>>>
> > >>>> My idea is, if 8 DIHs use 8 CPUs then I have 8 CPUs left for merging
> > >>>> (maxIndexingThreads/maxMergeAtOnce/mergeFactor).
> > >>>> It should do no commit, no optimize.
> > >>>> ramBufferSizeMB is high because I have plenty of RAM and I want
make
> > >> use the speed of RAM.
> > >>>> segmentsPerTier is high to reduce merging.
> > >>>>
> > >>>> But somewhere is a misconfiguration because indexing gets stalled.
> > >>>>
> > >>>> Any idea what's going wrong?
> > >>>>
> > >>>>
> > >>>> Bernd
> > >>>>
> > >>
> > >
> >
> > --
> > *************************************************************
> > Bernd Fehling                    Bielefeld University Library
> > Dipl.-Inform. (FH)                LibTec - Library Technology
> > Universitätsstr. 25                  and Knowledge Management
> > 33615 Bielefeld
> > Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de
> >
> > BASE - Bielefeld Academic Search Engine - www.base-search.net
> > *************************************************************
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message