lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@gmail.com>
Subject Re: Static index, fastest way to do forceMerge
Date Fri, 02 Nov 2018 21:17:03 GMT
Thanks for chipping in, Toke. A ~1TB index is impressive.

Back of the envelope says reading & writing 900GB in 8 hours is
2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our
SSD machine, but even with SATA II this is only ~1/5th of the possible
fairly sequential IO throughput. So for us at least, NVMe drives are
not needed to have single-threaded CPU as bottleneck.

The mileage will vary depending on the CPU -- if it can merge the data
from multiple files at ones fast enough then it may theoretically
saturate the bandwidth... but I agree we also seem to be CPU bound on
these N-to-1 merges, a regular SSD is enough.

> And +1 to the issue BTW.

I agree. Fine-grained granularity here would be a win even in the
regular "merge is a low-priority citizen" case. At least that's what I
tend to think. And if there are spare CPUs, the gain would be
terrific.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message