lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Troullis <cptroul...@gmail.com>
Subject Long blocking during indexing + deleteByQuery
Date Sun, 05 Nov 2017 19:20:55 GMT
Hi,

I am experiencing an issue where threads are blocking for an extremely long
time when I am indexing while deleteByQuery is also running.

Setup info:
-Solr Cloud 6.6.0
-Simple 2 Node, 1 Shard, 2 replica setup
-~12 million docs in the collection in question
-Nodes have 64 GB RAM, 8 CPUs, spinning disks
-Soft commit interval 10 seconds, Hard commit (open searcher false) 60
seconds
-Default merge policy settings (Which I think is 10/10).

We have a query heavy index heavyish use case. Indexing is constantly
running throughout the day and can be bursty. The indexing process handles
both updates and deletes, can spin up to 15 simultaneous threads, and sends
to solr in batches of 3000 (seems to be the optimal number per trial and
error).

I can build the entire collection from scratch using this method in < 40
mins and indexing is in general super fast (averages about 3 seconds to
send a batch of 3000 docs to solr). The issue I am seeing is when some
threads are adding/updating documents while other threads are issuing
deletes (using deleteByQuery), solr seems to get into a state of extreme
blocking on the replica, which results in some threads taking 30+ minutes
just to send 1 batch of 3000 docs. This collection does use child documents
(hence the delete by query _root_), not sure if that makes a difference, I
am trying to duplicate on a non-child doc collection. CPU/IO wait seems
minimal on both nodes, so not sure what is causing the blocking.

Here is part of the stack trace on one of the blocked threads on the
replica:

qtp592179046-576 (576)
java.lang.Object@608fe9b5
org.apache.solr.update.DirectUpdateHandler2.addAndDelete​(DirectUpdateHandler2.java:354)
org.apache.solr.update.DirectUpdateHandler2.addDoc0​(DirectUpdateHandler2.java:237)
org.apache.solr.update.DirectUpdateHandler2.addDoc​(DirectUpdateHandler2.java:194)
org.apache.solr.update.processor.RunUpdateProcessor.processAdd​(RunUpdateProcessorFactory.java:67)
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd​(UpdateRequestProcessor.java:55)
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd​(DistributedUpdateProcessor.java:979)
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd​(DistributedUpdateProcessor.java:1192)
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd​(DistributedUpdateProcessor.java:748)
org.apache.solr.handler.loader.JavabinLoader$1.update​(JavabinLoader.java:98)
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator​(JavaBinUpdateRequestCodec.java:180)
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator​(JavaBinUpdateRequestCodec.java:136)
org.apache.solr.common.util.JavaBinCodec.readObject​(JavaBinCodec.java:306)
org.apache.solr.common.util.JavaBinCodec.readVal​(JavaBinCodec.java:251)
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList​(JavaBinUpdateRequestCodec.java:122)
org.apache.solr.common.util.JavaBinCodec.readObject​(JavaBinCodec.java:271)
org.apache.solr.common.util.JavaBinCodec.readVal​(JavaBinCodec.java:251)
org.apache.solr.common.util.JavaBinCodec.unmarshal​(JavaBinCodec.java:173)
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal​(JavaBinUpdateRequestCodec.java:187)
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs​(JavabinLoader.java:108)
org.apache.solr.handler.loader.JavabinLoader.load​(JavabinLoader.java:55)
org.apache.solr.handler.UpdateRequestHandler$1.load​(UpdateRequestHandler.java:97)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody​(ContentStreamHandlerBase.java:68)
org.apache.solr.handler.RequestHandlerBase.handleRequest​(RequestHandlerBase.java:173)
org.apache.solr.core.SolrCore.execute​(SolrCore.java:2477)
org.apache.solr.servlet.HttpSolrCall.execute​(HttpSolrCall.java:723)
org.apache.solr.servlet.HttpSolrCall.call​(HttpSolrCall.java:529)

A cursory search lead me to this JIRA
https://issues.apache.org/jira/browse/SOLR-7836, not sure if related though.

Can anyone shed some light on this issue? We don't do deletes very
frequently, but it is bringing solr to it's knees when we do, which is
causing some big problems.

Thanks,

Chris

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message