lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anahita Shayesteh-SSI <anahita...@ssi.samsung.com>
Subject Lucene indexing speed on NVMe drive
Date Thu, 30 Apr 2015 18:25:15 GMT
Hi. I am studying Lucene performance and in particular how it benefits from faster I/O such
as SSD and NVMe.
I am using nightlybench for indexing wiki (1K docs) with similar parameters as used in nightlyBench.
(Hardware: Intel Xeon, 2.5GHz, 20 processor ,40 with hyperthreading, 64G Memory) and study
indexing speed on HDD, SSD and NVMe. While I do see benefit when switching from HDD to SSD,
there is not much noticeable benefit moving to NVMe.
I get best performance (200GB/hour) with 20 indexing threads, increasing number of threads
to 40 hurts performance. Similarly increasing maxConcurrentMerges above 3-5 doesn't seem to
give me any benefit. I am wondering what the bottleneck is, or anyone has insight on  set
of options (number of threads, merge options, flush options, read buffer?) to take advantage
of a very fast I/O system. I see NVMe bandwidth going as high as 800MB/s but it is only fast
spikes and CPU utilization is about 50% on average, though some cores have consistently higher
utilization while others have spiky behavior.
You thoughts and inside is greatly appreciated. Thanks.
Anahita Shayesteh


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message