lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Roustant <bruno.roust...@gmail.com>
Subject Re: High concurrency scenarios IndexReader‘s registerParentReader method takes a lot of CPU time
Date Fri, 17 Apr 2020 15:49:40 GMT
Hi,

Next time please reply/send these technical questions to
java-user@lucene.apache.org or dev@lucene.apache.org to get more audience.

538 segments seems huge. What is your configuration to get this number of
segments?

Le jeu. 16 avr. 2020 à 16:28, 张海雷 <zhanghailei@bytedance.com> a écrit :

>
>
>
> Hi, all
> In our Elastic Search benchmark, found out that  IndexReader‘s
> registerParentReader method takes a lot of CPU time
> Environment:
> ES Version: 7.5.1
> Lucene Version: 8.3.0
>
> Index:
> Shard Num 1
> Replicator’s Num:1
> Segment Count: 538
> Size:49.5GB
>
> Benchmark result: qps:5200,pct99 600ms
> CPU Util:93%
>
> Use flame graph find the hot method  IndexReader‘s registerParentReader.
> dive into code find out that Have a  race condition that on the
> synchronizedSet Because of only one
> Shard and one shard have too many segments, this intensify competition
>
>
>
> To verify this assumption,execute force merge on this index,after force
> merge:
> QPS: more then 5600
> PCT99: 60ms
> CPU Util: 11%
> The performance improved by a factor of 10x
>
> So we can use ConcurrentWeakKeyHashmap instead of synchronizedSet  to
> reduce lock competition?
>
>
>
>
>
>
>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message