lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Pendap <SPen...@walmartlabs.com>
Subject Solr Query Performance benchmarking
Date Thu, 27 Apr 2017 23:20:53 GMT
Hi,
I am trying to perform Solr Query performance benchmarking and trying to measure the maximum
throughput and latency that I can get from.a given Solr cluster.

Following are my configurations

Number of Solr Nodes: 4
Number of shards: 2
replication-factor:  2
Index size: 55 GB
Shard/Core size: 27.7 GB
maxConnsPerHost: 1000

The Solr nodes are VM's with 16 core vCpu and 112GB RAM.  The CPU is 1-1 and it is not overcommitted.

I am generating query load using a Java client program which fires Solr queries read from
a static file.  The client java program is using the Apache Http Client library to invoke
the queries. I have already configured the client to create 300 max connections.

The type of queries are mostly of the below pattern
q=*:*&fl=orderNo,purchaseOrderNos,timestamp,eventName,eventID,_src_&fq=((orderNo:<orderNoValue>+AND+purchaseOrderNos:<purchaseOrderNoValue)+OR+(+orderNo:<orderNoValue)+OR+(+purchaseOrderNos:<purchaseOrderNoValue>))&sort=eventTimestamp+desc&rows=20&wt=javabin&version=2

Max throughput that I get: 12000 to 12500 reqs/sec
95 percentile query latency: 30 to 40 msec

I am measuring the latency and throughput on the client side in my program.  The max throughput
that I am able to get (sum of each individual clients throughput) is 12000 reqs/sec.  I am
running with 4 clients each with 50 threads.  Even if I increase the number of clients, the
throughput still seems to be the same. It seems like I am hitting the maximum capacity of
the cluster or some other limit due to which I am not able to put more stress on the server.

My CPU is hitting 60% to 70%.  I have not been able to increase the CPU usage more than this
even when increasing client threads or generating load with more client nodes.

The memory used is around 16% on all the nodes except on one node I am seeing the memory used
is 41%.

There is hardly any IO happening as it is a read test.

I am wondering what is limiting my throughput, is there some internal thread pool limit that
I am hitting due to which I am not able to increase my CPU/memory usage?

My JVM settings are provided below. I am using G1GC and


-DSTOP.KEY=solrrocks
-DSTOP.PORT=7983
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=13001
-Dcom.sun.management.jmxremote.rmi.port=13001
-Dcom.sun.management.jmxremote.ssl=false
-Djetty.home=/app/solr6/server
-Djetty.port=8983
-Dlog4j.configuration=file:<log4j.properties file>
-Dsolr.autoSoftCommit.maxTime=5000
-Dsolr.autoSoftCommit.minTime=5000
-Dsolr.install.dir=/app/solr6
-Dsolr.log.dir=/app/solrdata6/logs
-Dsolr.log.muteconsole
-Dsolr.solr.home=<solr home dir>
-Duser.timezone=UTC
-DzkClientTimeout=15000
-DzkHost=<ZkHostName>
-XX:+AlwaysPreTouch
-XX:+ResizeTLAB
-XX:+UseG1GC
-XX:+UseGCLogFileRotation
-XX:+UseLargePages
-XX:+UseTLAB
-XX:-UseBiasedLocking
-XX:GCLogFileSize=20M
-XX:MaxGCPauseMillis=50
-XX:NumberOfGCLogFiles=9
-XX:OnOutOfMemoryError=/app/solr6/bin/oom_solr.sh
-Xloggc:<solr gc log file>
-Xms11g
-Xmx11g
-Xss256k
-verbose:gc
















I have not customized the Solr Cache values.  The DocumentCache, QueryResultCache, FieldValueCache
everything is using default values.  I read in one of the SolrPerformance documents that it
is better to leave more memory to the Operating system and utilize the OS buffer cache.

Is it the best query throughput that I can extract from this sized cluster and index size
combination?

Any ideas is highly appreciated.

Thanks
Suresh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message