lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Joyner <mich...@newsrx.com>
Subject Re: Solr Cloud 6.5.0 Replicas go down while indexing
Date Tue, 04 Apr 2017 14:02:03 GMT
Try Increasing the number of connections your ZooKeeper allows to a very 
large number.


On 04/04/2017 09:02 AM, Salih Sen wrote:
> Hi,
>
> One of the replicas went down again today somehow disabling all 
> updates to cluster with error message "Cannot talk to ZooKeeper - 
> Updates are disabled.” half an hour.
>
> ZK Leader was on the same server with Solr instance so I doubt it has 
> anything to do with network (at least between Solr and ZK leader 
> node), restarting the ZK leader seems to resolve the issue and cluster 
> accepting updates again.
>
>
> == Solr Node
> WARN  - 2017-04-04 11:49:14.414; [   ] 
> org.apache.solr.common.cloud.ConnectionManager; Watcher 
> org.apache.solr.common.cloud.ConnectionManager@44ca0f2f name: 
> ZooKeeperConnection Watcher:192.168.30.32:2181 
> <http://192.168.30.32:2181>,192.168.30.33:2181 
> <http://192.168.30.33:2181>,192.168.30.24:2181 
> <http://192.168.30.24:2181> got event WatchedEvent state:Disconnected 
> type:None path:null path: null type: None
> WARN  - 2017-04-04 11:49:15.723; [   ] 
> org.apache.solr.common.cloud.ConnectionManager; zkClient has disconnected
> WARN  - 2017-04-04 11:49:15.727; [   ] 
> org.apache.solr.common.cloud.ConnectionManager; Watcher 
> org.apache.solr.common.cloud.ConnectionManager@44ca0f2f name: 
> ZooKeeperConnection Watcher:192.168.30.32:2181 
> <http://192.168.30.32:2181>,192.168.30.33:2181 
> <http://192.168.30.33:2181>,192.168.30.24:2181 
> <http://192.168.30.24:2181> got event WatchedEvent state:Expired 
> type:None path:null path: null type: None
> WARN  - 2017-04-04 11:49:15.727; [   ] 
> org.apache.solr.common.cloud.ConnectionManager; Our previous ZooKeeper 
> session was expired. Attempting to reconnect to recover relationship 
> with ZooKeeper...
> WARN  - 2017-04-04 11:49:15.728; [   ] 
> org.apache.solr.common.cloud.DefaultConnectionStrategy; Connection 
> expired - starting a new one...
> ERROR - 2017-04-04 11:49:22.040; [c:doc s:shard6 r:core_node27 
> x:doc_shard6_replica1] org.apache.solr.common.SolrException; 
> org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - 
> Updates are disabled.
>         at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.zkCheck(DistributedUpdateProcessor.java:1739)
>         at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:703)
>         at 
> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:97)
>         at 
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:179)
>         at 
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:135)
>         at 
> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:306)
>         at 
> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:251)
>         at 
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
>         at 
> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:271)
>         at 
> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:251)
>         at 
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:173)
>         at 
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:186)
>         at 
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
>         at 
> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)
>         at 
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
>         at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>         at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
>         at 
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
>         at 
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
>         at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
>         at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
>         at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
>         at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>         at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>         at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>         at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>         at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at 
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>         at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:534)
>
> ERROR - 2017-04-04 11:50:13.798; [   ] 
> org.apache.solr.common.SolrException; 
> null:org.apache.solr.common.SolrException: Error trying to proxy 
> request for url: http://192.168.30.24:9141/solr/doc/select
>         at 
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:659)
>         at 
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:513)
>         at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
>         at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
>         at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
>         at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>         at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>         at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>         at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>         at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at 
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>         at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:534)
>         at 
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>         at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>         at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at 
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
>         at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
>         at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.eclipse.jetty.io.EofException
>         at 
> org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:199)
>         at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:420)
>         at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:313)
>         at 
> org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:140)
>         at 
> org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:744)
>         at 
> org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
>         at 
> org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
>         at 
> org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:518)
>         at 
> org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:724)
>         at 
> org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:775)
>         at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:235)
>         at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:219)
>         at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:496)
>         at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2147)
>         at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)
>         at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)
>         at 
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:655)
>
>
> === ZK Leader
> 2017-04-04 14:48:46,327 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197>] - Accepted 
> socket connection from /192.168.30.24:57990 <http://192.168.30.24:57990>
> 2017-04-04 14:48:46,499 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197>] - Accepted 
> socket connection from /192.168.30.24:57994 <http://192.168.30.24:57994>
> 2017-04-04 14:48:50,005 [myid:3] - INFO 
>  [SessionTracker:ZooKeeperServer@347] - Expiring session 
> 0x15b14ba8a8e0054, timeout of 40000ms exceeded
> 2017-04-04 14:48:50,005 [myid:3] - INFO  [ProcessThread(sid:3 
> cport:-1)::PrepRequestProcessor@494] - Processed session termination 
> for sessionid: 0x15b14ba8a8e0054
> 2017-04-04 14:48:59,821 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861 
> <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861>] - Client attempting 
> to renew session 0x15b14ba8a8e004b at /192.168.30.24:57990 
> <http://192.168.30.24:57990>
> 2017-04-04 14:48:59,822 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@617 
> <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@617>] - Established 
> session 0x15b14ba8a8e004b with negotiated timeout 40000 for client 
> /192.168.30.24:57990 <http://192.168.30.24:57990>
> 2017-04-04 14:48:59,822 [myid:3] - WARN 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357>] - caught end of 
> stream exception
> EndOfStreamException: Unable to read additional data from client 
> sessionid 0x15b14ba8a8e004b, likely client has closed socket
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>         at 
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-04-04 14:48:59,827 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007>] - Closed socket 
> connection for client /192.168.30.24:57990 
> <http://192.168.30.24:57990> which had sessionid 0x15b14ba8a8e004b
> 2017-04-04 14:48:59,827 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861 
> <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861>] - Client attempting 
> to renew session 0x15b14ba8a8e0050 at /192.168.30.24:57994 
> <http://192.168.30.24:57994>
> 2017-04-04 14:48:59,827 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@617 
> <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@617>] - Established 
> session 0x15b14ba8a8e0050 with negotiated timeout 40000 for client 
> /192.168.30.24:57994 <http://192.168.30.24:57994>
> 2017-04-04 14:49:17,455 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197>] - Accepted 
> socket connection from /192.168.30.24:58082 <http://192.168.30.24:58082>
> 2017-04-04 14:49:17,667 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197>] - Accepted 
> socket connection from /192.168.30.32:56600 <http://192.168.30.32:56600>
> 2017-04-04 14:49:17,667 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861 
> <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861>] - Client attempting 
> to renew session 0x15b14ba8a8e0043 at /192.168.30.32:56600 
> <http://192.168.30.32:56600>
> 2017-04-04 14:49:17,681 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@617 
> <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@617>] - Established 
> session 0x15b14ba8a8e0043 with negotiated timeout 40000 for client 
> /192.168.30.32:56600 <http://192.168.30.32:56600>
> 2017-04-04 14:49:22,040 [myid:3] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868 
> <http://0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868>] - Client attempting 
> to establish new session at /192.168.30.24:58082 
> <http://192.168.30.24:58082>
> 2017-04-04 14:49:22,051 [myid:3] - INFO 
>  [CommitProcessor:3:ZooKeeperServer@617] - Established session 
> 0x35ad61c452c00d3 with negotiated timeout 40000 for client 
> /192.168.30.24:58082 <http://192.168.30.24:58082>
> 2017-04-04 14:49:28,659 [myid:3] - INFO  [ProcessThread(sid:3 
> cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException 
> when processing sessionid:0x35ad61c452c00d3 type:delete cxid:0xe 
> zxid:0x700004c25 txntype:-1 reqpath:n/a Error 
> Path:/overseer_elect/election/97694608339632212-192.168.30.24:9133_solr-n_0000000380

> Error:KeeperErrorCode = NoNode for 
> /overseer_elect/election/97694608339632212-192.168.30.24:9133_solr-n_0000000380
> 2017-04-04 14:49:28,675 [myid:3] - INFO  [ProcessThread(sid:3 
> cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException 
> when processing sessionid:0x35ad61c452c00d3 type:create cxid:0x13 
> zxid:0x700004c27 txntype:-1 reqpath:n/a Error Path:/overseer 
> Error:KeeperErrorCode = NodeExists for /overseer
>
>
> == ZK Follower 1
> 2017-04-04 14:48:45,570 [myid:1] - WARN 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357>] - caught end of 
> stream exception
> EndOfStreamException: Unable to read additional data from client 
> sessionid 0x15b14ba8a8e004b, likely client has closed socket
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>         at 
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-04-04 14:48:45,587 [myid:1] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007>] - Closed socket 
> connection for client /192.168.30.24:39820 
> <http://192.168.30.24:39820> which had sessionid 0x15b14ba8a8e004b
> 2017-04-04 14:48:45,587 [myid:1] - WARN 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357>] - caught end of 
> stream exception
> EndOfStreamException: Unable to read additional data from client 
> sessionid 0x15b14ba8a8e0050, likely client has closed socket
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>         at 
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-04-04 14:48:45,589 [myid:1] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007>] - Closed socket 
> connection for client /192.168.30.24:40132 
> <http://192.168.30.24:40132> which had sessionid 0x15b14ba8a8e0050
> 2017-04-04 14:48:48,351 [myid:1] - WARN 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357>] - caught end of 
> stream exception
> EndOfStreamException: Unable to read additional data from client 
> sessionid 0x15b14ba8a8e0054, likely client has closed socket
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>         at 
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-04-04 14:48:48,352 [myid:1] - INFO 
>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007 
> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007>] - Closed socket 
> connection for client /192.168.30.24:40212 
> <http://192.168.30.24:40212> which had sessionid 0x15b14ba8a8e0054
> 2017-04-04 15:24:03,034 [myid:1] - WARN 
>  [RecvWorker:3:QuorumCnxManager$RecvWorker@780] - Connection broken 
> for id 3, my id = 1, error =
> java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at 
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:765)
> 2017-04-04 15:24:03,053 [myid:1] - WARN 
>  [RecvWorker:3:QuorumCnxManager$RecvWorker@783] - Interrupting SendWorker
> 2017-04-04 15:24:03,093 [myid:1] - WARN 
>  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@89] - Exception 
> when following the leader
> java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at 
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
>         at 
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
>         at 
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
>         at 
> org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
>         at 
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)
>         at 
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786)
>
>
>
>
>
> *Salih Şen*
> M: +90 533 131 17 07
> E: salih@dilisim.com <mailto:salih@dilisim.com>
> W: www.dilisim.com <http://www.dilisim.com>
> Skype: slhsen
>
> On 4 April 2017 at 10:36:14, Salih Sen (salih@dilisim.com 
> <mailto:salih@dilisim.com>) wrote:
>
>> Hi,
>>
>> Sorry for the initial hurried up mail, here is some correction and 
>> further explanation:
>>
>> Problem I described previously was happening before we set 
>> zkClientTimeout value so it was 30000 when it happened.
>>
>> autoCommit maxTime value is 15000 and autoSoftCommit maxTime is 60000.
>>
>> We recently removed maxDocs values from autoCommit settings and it 
>> seems more stable so far and has better response time.
>>
>> I can’t seem to find these values on Solr logs probably because 
>> logging level is currently WARN but we left those as default so I 
>> think they’re set as the values in solr.xml
>> <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int>
>> <int 
>> name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int>
>>
>>
>> We have 12 replicas using default routing. All commits and queries 
>> are going to a single node because of the dummy client we use. 
>> Documents are send in JSON format. I don’t have exact knowledge of 
>> document size, they are mostly news article sized, though with lots 
>> of dynamic fields.
>>
>> Sematext SPM currently shows “Added Docs Rate” as ~1.70k/sec for the 
>> server that is receiving updates.
>>
>> Once problem starts happening multiple replicas go down (not 
>> necessarily the one receiving the update request from client) and 
>> cluster starts returning errors to update requests.
>>
>>
>> We saw entries like following in Zookeeper logs that’s why we thought 
>> it might be related to zkClientTimeout and value.
>>
>> 2017-04-03 09:13:03,040 [myid:1] - INFO 
>>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007 
>> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007>] - Closed socket 
>> connection for client /192.168.30.32:36420 
>> <http://192.168.30.32:36420> which had sessionid 0x25ad61c4507008c
>> 2017-04-03 09:27:02,078 [myid:1] - WARN 
>>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357 
>> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357>] - caught end of 
>> stream exception
>> EndOfStreamException: Unable to read additional data from client 
>> sessionid 0x15b14ba8a8e0026, likely client has closed socket
>> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>> at 
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>> at java.lang.Thread.run(Thread.java:745)
>> 2017-04-03 09:27:02,079 [myid:1] - INFO 
>>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007 
>> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007>] - Closed socket 
>> connection for client /192.168.30.32:35636 
>> <http://192.168.30.32:35636> which had sessionid 0x15b14ba8a8e0026
>> 2017-04-03 09:35:19,362 [myid:1] - INFO 
>>  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197 
>> <http://0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197>] - Accepted 
>> socket connection from /192.168.30.32:37970 <http://192.168.30.32:37970>
>>
>>
>>
>> *Salih Şen*
>> M: +90 533 131 17 07
>> E: salih@dilisim.com <mailto:salih@dilisim.com>
>> W: www.dilisim.com <http://www.dilisim.com>
>> Skype: slhsen
>>
>> On 3 April 2017 at 18:01:15, Erick Erickson (erickerickson@gmail.com 
>> <mailto:erickerickson@gmail.com>) wrote:
>>
>>> bq: We set Auto hardcommit time to 15sec and 10000 docs, and soft
>>> commit to 60000 sec and 5000 seconds
>>>
>>> Just a sanity check, the commit intervals are in milliseconds, your
>>> units look mixed up above, I'm guessing it's just a typo though. I
>>> usually don't use maxDocs because it's unpredictable. Say you're
>>> indexing at a furious rate. If you are indexing at 5,000 docs a second
>>> (and assuming the above was supposed to be soft committing every 60
>>> seconds or 5,000 docs) you'll still be autocommitting every second.
>>>
>>> While that could be related, it's not particularly germane to your
>>> timeout. My guess is that you're getting these errors on the leader?
>>> what do you have in solr.xml for:
>>>
>>> distribUpdateConnTimeout and distribUpdateSoTimeout
>>>
>>> Those are likely the timeouts that matter. And how big are your
>>> documents? The scenario I'm thinking of is that the leader sends the
>>> update to the replica and the timeout for the replica's response
>>> exceeds the ones above.
>>>
>>> BTW, it can be useful on startup to look at your solr.log. The
>>> _actual_ values for all the timeouts are printed out, including any
>>> sysvars you've used.
>>>
>>> And how are you indexing? Mostly I'm wondering how fast you're sending
>>> docs to each leader and how.
>>>
>>> Best,
>>> Erick
>>>
>>> On Mon, Apr 3, 2017 at 6:52 AM, Salih Sen <salih@dilisim.com 
>>> <mailto:salih@dilisim.com>> wrote:
>>> > Hi,
>>> >
>>> > We have a three server set up with each server having 756G ram, 48 
>>> cores,
>>> > 4SSDs (each having tree solr instances on them) and a dedicated 
>>> mechanical
>>> > disk for zookeeper (3 zk instances total). Each Solr instances 
>>> have 31G of
>>> > heap space allocated to them. In total we have 36 Solr Instances and 3
>>> > Zookeeper instances (with 1G heapspace). Also servers 10Gig 
>>> network between
>>> > them.
>>> >
>>> > We set Auto hardcommit time to 15sec and 10000 docs, and soft 
>>> commit to
>>> > 60000 sec and 5000 seconds in order to avoid soft committing too 
>>> much and
>>> > avoiding indexing bottlenecks. We also set DzkClientTimeout=90000.
>>> >
>>> > But it seems replicas still randomly go down while indexing. Do 
>>> you have any
>>> > suggestions to prevent this situation?
>>> >
>>> > ERROR - 2017-04-03 12:24:02.503; [ ]
>>> > org.apache.solr.cloud.OverseerCollectionMessageHandler; Error from 
>>> shard:
>>> > http://192.168.30.33:9132/solr
>>> > org.apache.solr.client.solrj.SolrServerException: Timeout occured 
>>> while
>>> > waiting response from server at: http://192.168.30.33:9132/solr
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:621)
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
>>> > at 
>>> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
>>> > at
>>> > 
>>> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:163)
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> > at 
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> > at
>>> > 
>>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
>>> > at
>>> > 
>>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>>> > at
>>> > 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> > at
>>> > 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> > at java.lang.Thread.run(Thread.java:745)
>>> > Caused by: java.net.SocketTimeoutException: Read timed out
>>> > at java.net.SocketInputStream.socketRead0(Native Method)
>>> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>>> > at java.net.SocketInputStream.read(SocketInputStream.java:171)
>>> > at java.net.SocketInputStream.read(SocketInputStream.java:141)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>>> > at
>>> > 
>>> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>>> > at
>>> > 
>>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>>> > at
>>> > 
>>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
>>> > at
>>> > 
>>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>>> > at
>>> > 
>>> org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
>>> > at
>>> > 
>>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
>>> > at
>>> > 
>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>>> > at
>>> > 
>>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>>> > at
>>> > 
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>>> > at
>>> > 
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
>>> > ... 12 more
>>> > ERROR - 2017-04-03 12:27:11.631; [c:doc s:shard3 r:core_node22
>>> > x:doc_shard3_replica3]
>>> > 
>>> org.apache.solr.update.StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient;
>>> > error
>>> > java.net.SocketTimeoutException: Read timed out
>>> > at java.net.SocketInputStream.socketRead0(Native Method)
>>> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>>> > at java.net.SocketInputStream.read(SocketInputStream.java:171)
>>> > at java.net.SocketInputStream.read(SocketInputStream.java:141)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>>> > at
>>> > 
>>> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>>> > at
>>> > 
>>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>>> > at
>>> > 
>>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
>>> > at
>>> > 
>>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>>> > at
>>> > 
>>> org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
>>> > at
>>> > 
>>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
>>> > at
>>> > 
>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>>> > at
>>> > 
>>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>>> > at
>>> > 
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>>> > at
>>> > 
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
>>> > at
>>> > 
>>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
>>> > at
>>> > 
>>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>>> > at
>>> > 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> > at
>>> > 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> > at java.lang.Thread.run(Thread.java:745)
>>> > ERROR - 2017-04-03 12:27:11.633; [c:doc s:shard3 r:core_node22
>>> > x:doc_shard3_replica3]
>>> > 
>>> org.apache.solr.update.StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient;
>>> > error
>>> > java.net.SocketTimeoutException: Read timed out
>>> > at java.net.SocketInputStream.socketRead0(Native Method)
>>> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>>> > at java.net.SocketInputStream.read(SocketInputStream.java:171)
>>> > at java.net.SocketInputStream.read(SocketInputStream.java:141)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>>> > at
>>> > 
>>> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>>> > at
>>> > 
>>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>>> > at
>>> > 
>>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
>>> > at
>>> > 
>>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>>> > at
>>> > 
>>> org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
>>> > at
>>> > 
>>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
>>> > at
>>> > 
>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>>> > at
>>> > 
>>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>>> > at
>>> > 
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>>> > at
>>> > 
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
>>> > at
>>> > 
>>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
>>> > at
>>> > 
>>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>>> > at
>>> > 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> > at
>>> > 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> > at java.lang.Thread.run(Thread.java:745)
>>> > ERROR - 2017-04-03 12:27:11.645; [c:doc s:shard3 r:core_node22
>>> > x:doc_shard3_replica3]
>>> > 
>>> org.apache.solr.update.StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient;
>>> > error
>>> > java.net.SocketTimeoutException: Read timed out
>>> > at java.net.SocketInputStream.socketRead0(Native Method)
>>> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>>> > at java.net.SocketInputStream.read(SocketInputStream.java:171)
>>> > at java.net.SocketInputStream.read(SocketInputStream.java:141)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>>> > at
>>> > 
>>> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>>> > at
>>> > 
>>> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>>> > at
>>> > 
>>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
>>> > at
>>> > 
>>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>>> > at
>>> > 
>>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
>>> > at
>>> > 
>>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>>> > at
>>> > 
>>> org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
>>> > at
>>> > 
>>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
>>> > at
>>> > 
>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>>> > at
>>> > 
>>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>>> > at
>>> > 
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>>> > at
>>> > 
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
>>> > at
>>> > 
>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
>>> > at
>>> > 
>>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
>>> > at
>>> > 
>>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>>> > at
>>> > 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> > at
>>> > 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> > at java.lang.Thread.run(Thread.java:745)
>>> > W
>>> >
>>> >
>>> >
>>> >
>>> > Salih Şen
>>> > M: +90 533 131 17 07
>>> > E: salih@dilisim.com <mailto:salih@dilisim.com>
>>> > W: www.dilisim.com <http://www.dilisim.com>
>>> > Skype: slhsen



Mime
View raw message