lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nawab Zada Asad Iqbal <>
Subject Understanding flush and DocumentsWriterPerThread
Date Sat, 05 Aug 2017 19:59:31 GMT

I am debugging a bulk indexing performance issue while upgrading to 6.6
from 4.5.0 . I have commits disabled while indexing total of 85G data
during 7 hours. At the end of it, I want some 30 or so big segments. But i
am getting 3000 segments.
I deleted the index and enabled infostream logging ; i have attached the
log when first segment is flushed. Here are few questions:

1. When a segment if flushed , then is it permanent or can more documents
be written to it (besides the merge scenario)?
2. It seems that 330+ threads are writing in parallel. Will each one of
them become one segment when written to the disk? In which case, i should
probably decrease concurrency?
3. One possibility is to delay flushing, the flush is getting triggered at
10000MB, probably coming from <ramBufferSizeMB>10000</ramBufferSizeMB> ;
however, the segment which is flushed is only 115MB. Is this limit for the
combined size of all in-memory segments? In which case, is it ok to
increase it further to use more of my heap (48GB).
4. How can I decrease the concurrency, maybe the solution is to use fewer
in memory segments?

In previous run, there were 110k files in the index folder after I stopping
indexing. Before doing commit, I noticed that the file count continued to
decrease every few minutes, until it reduced to 27k or so. (I committed
after it stabilized)

My Indexconfig is this:

  <int name="maxMergeAtOnce">5</int>
     <int name="segmentsPerTier">3000</int>
      <int name="maxMergeAtOnceExplicit">10</int>
      <int name="floorSegmentMB">16</int>
      <!-- 200 gb since we want few big segments during full indexing -->
      <double name="maxMergedSegmentMB">200000</double>
      <double name="forceMergeDeletesPctAllowed">1</double>
       <int name="maxThreadCount">10</int>
       <int name="maxMergeCount">10</int>
    <deletionPolicy class="solr.SolrDeletionPolicy">
      <str name="maxCommitsToKeep">1</str>
      <str name="maxOptimizedCommitsToKeep">0</str>


View raw message