lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen <t...@kb.dk>
Subject Re: Static index, fastest way to do forceMerge
Date Fri, 02 Nov 2018 20:24:41 GMT
Dawid Weiss <dawid.weiss@gmail.com> wrote:
> Merging segments as large as this one requires not just CPU, but also
> serious I/O throughput efficiency. I assume you have fast NVMe drives
> on that machine, otherwise it'll be slow, no matter what. It's just a
> lot of bytes going back and forth.

We have quite a lot of experience with creating fully merged 900GB indexes. On our plain-SSD
(Samsung 840) equipped machine this took ~8 hours with a single CPU-core at 100%. On our 7200
RPM spinning drive machines (same CPU class) it took nearly twice as long. Back of the envelope
says reading & writing 900GB in 8 hours is 2*900GB/(8*60*60s) = 64MB/s. I don't remember
the interface for our SSD machine, but even with SATA II this is only ~1/5th of the possible
fairly sequential IO throughput. So for us at least, NVMe drives are not needed to have single-threaded
CPU as bottleneck.

And +1 to the issue BTW. It does not matter too much for us now, as we have shifted to a setup
where we build more indexes in parallel, but 3 years ago our process was sequential so the
8 hour delay before building the next part was a bit of an annoyance.

- Toke Eskildsen

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message