lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bhalla, Rahat" <rbha...@HealthPlan.com>
Subject RE: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
Date Wed, 05 Jul 2017 08:05:12 GMT
Hi

I'm not sure if any of you have had a chance to see this email yet.

We had a reoccurrence of the Issue Today, and I'm attaching the Logs from today as well inline
below.

Please let me know if any of you have seen this issue before as this would really help me
to get to the root of the problem to fix it. I'm a little lost here and not entirely sure
what to do.

Thanks,
Rahat Bhalla

8696248 [qtp778720569-28] [ WARN] 2017-07-04 01:40:20 (HttpParser.java:parseNext:1391) - parse
exception: java.lang.IllegalArgumentException: No Authority for HttpChannelOverHttp@30a86e14{r=0,c=false,a=IDLE,uri=null}
java.lang.IllegalArgumentException: No Authority
        at org.eclipse.jetty.http.HostPortHttpField.<init>(HostPortHttpField.java:43)
        at org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)
        at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)
        at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
8697308 [qtp778720569-21] [ WARN] 2017-07-04 01:40:21 (HttpParser.java:parseNext:1364) - bad
HTTP parsed: 400 Bad URI for HttpChannelOverHttp@127bbbb6{r=16,c=false,a=IDLE,uri=/../../../../../../../../../../etc/passwd}
8697338 [qtp778720569-29] [ WARN] 2017-07-04 01:40:21 (HttpParser.java:parseNext:1364) - bad
HTTP parsed: 400 No Host for HttpChannelOverHttp@50a994ce{r=29,c=false,a=IDLE,uri=null}
8697388 [qtp778720569-21] [ WARN] 2017-07-04 01:40:22 (HttpParser.java:parseNext:1364) - bad
HTTP parsed: 400 Bad URI for HttpChannelOverHttp@19a624ec{r=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd}
8697401 [qtp778720569-27] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte C0 in state 0
8697444 [qtp778720569-25] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte 80 in state 4
8697475 [qtp778720569-26] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte 80 in state 6
8697500 [qtp778720569-29] [ WARN] 2017-07-04 01:40:22 (URIUtil.java:decodePath:348) - /solr/admin/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! byte F8 in state 0
8706641 [qtp778720569-27] [ WARN] 2017-07-04 01:40:31 (HttpParser.java:parseNext:1364) - bad
HTTP parsed: 400 Unknown Version for HttpChannelOverHttp@7fcd594a{r=54,c=false,a=IDLE,uri=null}
8707033 [qtp778720569-20] [ WARN] 2017-07-04 01:40:31 (HttpParser.java:parseNext:1364) - bad
HTTP parsed: 400 Unknown Version for HttpChannelOverHttp@66740d77{r=54,c=false,a=IDLE,uri=null}
8719390 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 (HttpParser.java:<init>:1740)
- Illegal character 0xA in state=HEADER_IN_NAME for buffer HeapByteBuffer@41ed5e30[p=220,l=273,c=8192,r=53]={POST
/wp-content/...name="qqfile"\r\n<<<filename="QualysU...ys-Scan: VM\r\n\r\n>>>\n\r\nq=*%3A*&df=tex...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
8719391 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 (HttpParser.java:parseNext:1364) - bad
HTTP parsed: 400 Illegal character 0xA for HttpChannelOverHttp@235b51f6{r=2,c=false,a=IDLE,uri=null}
8949147 [qtp778720569-27] [ WARN] 2017-07-04 01:44:33 (HttpParser.java:parseNext:1364) - bad
HTTP parsed: 400 HTTP/0.9 not supported for HttpChannelOverHttp@62d6261e{r=0,c=false,a=IDLE,uri=null}
8949149 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 (HttpParser.java:parseNext:1364) - bad
HTTP parsed: 400 HTTP/0.9 not supported for HttpChannelOverHttp@33f62c0{r=0,c=false,a=IDLE,uri=null}
8949163 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 (HttpParser.java:parseNext:1364) - bad
HTTP parsed: 400 HTTP/0.9 not supported for HttpChannelOverHttp@6d58f7d7{r=0,c=false,a=IDLE,uri=null}
67338023 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158]
[ERROR] 2017-07-04 05:57:42 (SolrException.java:log:159) - :org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)
        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
        at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299)
        at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:279)
        at org.apache.solr.cloud.DistributedQueue.peekElements(DistributedQueue.java:328)
        at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:236)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:226)
        at java.lang.Thread.run(Unknown Source)

67365997 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158]
[ERROR] 2017-07-04 05:58:10 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
67381954 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 (SolrException.java:log:148) - java.io.IOException:
Error opening /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more

67381955 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 (SolrException.java:log:159) - null:java.io.IOException:
Error opening /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /configs/feedsInBoundCarrier/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more

67393217 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158]
[ERROR] 2017-07-04 05:58:37 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
67420293 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158]
[ERROR] 2017-07-04 05:59:04 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
67437164 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 (SolrException.java:log:148) - java.io.IOException:
Error opening /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more

67437165 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 (SolrException.java:log:159) - null:java.io.IOException:
Error opening /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)
        at org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /configs/txnInBoundNonEx/db-data-config.xml
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
        at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
        at org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)
        at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)
        ... 32 more



From: Bhalla, Rahat
Sent: Monday, July 03, 2017 8:38 PM
To: solr-user@lucene.apache.org
Subject: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
Importance: High


Hi Solr Users,

I hope this email finds you all in the best of spirits and in a mood where you'd be willing
to help a young developer (me :) ) with issues that I'm facing in regards with the Solr Cloud.

At my organization, we are running a Solr Cloud with 5 Nodes for Solr Instances with 13 collections
spread across the 5 nodes and an ensemble of 3 zookeeper instances spread across three different
nodes.

Over the last one week, our leader node seems to be going down every other day and while we
restart the solr instances they still go down within the next 24 Hours or more.

We have tried rebooting the nodes that host the solr instances and that hasn't helped. We
plan to clear out the zookeeper logs and data folders before the restart of the zookeeper
instances.

As of now, I'm the only one supporting Solr in my organization and any insight from you could
help me a great deal to fix the issue. I'm copying the Exception stack trace from this morning.
Any recommendations that you might have will be great appreciated.

Below is a snapshot of one of the zoo nodes:

[cid:image001.png@01D2F543.E5447AD0]

Exception Stacktrace

138127149 [OverseerCollectionConfigSetProcessor-98234161688412161-prod-solr-node01:9080_solr-n_0000000140]
[ERROR] 2017-07-03 05:02:55 (OverseerTaskProcessor.java:amILeader:392) -
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
for /overseer_elect/leader
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)
        at java.lang.Thread.run(Unknown Source)
138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01 (SolrException.java:log:148) -
org.apache.solr.common.SolrException: Could not load collection from ZK: feedsOutBoundToExchange
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1047)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:610)
        at org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:248)
        at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation$20.call(CollectionsHandler.java:674)
        at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:195)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /collections/feedsOutBoundToExchange/state.json
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
        at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1059)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1045)
        ... 33 more

Thanks,

Rahat Bhalla
HealthPlan Services
Phone: (813) 289-1000  EXT: 7002249
rbhalla@healthplan.com<mailto:rbhalla@healthplan.com>
www.healthplan.com<http://www.healthplan.com/>


_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
CONFIDENTIALITY NOTICE: This email message, including any attachments, is for the sole use
of the intended recipient(s) and may contain confidential and privileged information and/or
Protected Health Information (PHI) subject to protection under the law, including the Health
Insurance Portability and Accountability Act of 1996, as amended (HIPAA). If you are not the
intended recipient or the person responsible for delivering the email to the intended recipient,
be advised that you have received this email in error and that any use, disclosure, distribution,
forwarding, printing, or copying of this email is strictly prohibited. If you have received
this email in error, please notify the sender immediately and destroy all copies of the original
message.
Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message