lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anshum Gupta <ans...@anshumgupta.net>
Subject Re: SPLITSHARD failed to invoke SPLIT core admin command
Date Fri, 03 Jan 2014 23:40:20 GMT
Also, here's another issue that would help with the timeouts
(SOLR-5477<https://issues.apache.org/jira/browse/SOLR-5477>
).


On Fri, Jan 3, 2014 at 3:34 PM, Anshum Gupta <anshum@anshumgupta.net> wrote:

> Hi Sriram,
>
> People hit timeout issues when trying to split larger collections/shards.
> This is because of the timing out of internal calls.
>
> About the UI showing everything as active, that's another open issue(
> SOLR-4777 <https://issues.apache.org/jira/browse/SOLR-4777>). You should
> instead check the clusterstate to see the state of the slices (both parent
> and child).
> What is the current state of the slices?
>
>
>
> On Fri, Jan 3, 2014 at 2:39 PM, sriram <Sriram.V@ask.com> wrote:
>
>> Hi,
>>
>> I am using solr cloud 4.6 with 2 shards. I have loaded close to 13Million
>> data in them. When I tried to give split shard, I am getting the following
>> errors in the log,
>>
>> INFO  - 2014-01-03 14:17:45.357;
>> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; Watcher fired on
>> path: /overseer/collection-queue-work/qnr-0000000002 state: SyncConnected
>> type NodeDataChanged
>> INFO  - 2014-01-03 14:17:45.359;
>> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; Watcher fired on
>> path: /overseer/collection-queue-work state: SyncConnected type
>> NodeChildrenChanged
>> INFO  - 2014-01-03 14:17:45.360;
>> org.apache.solr.cloud.OverseerCollectionProcessor; Overseer Collection
>> Processor: Message id:/overseer/collection-queue-work/qn-0000000002
>> complete,
>>
>> response:{success={null={responseHeader={status=0,QTime=3689},core=collection1_shard1_1_replica1},null={responseHeader={status=0,QTime=3694},core=collection1_shard1_0_replica1},null={responseHeader={status=0,QTime=2004}},null={responseHeader={status=0,QTime=2005}}},failure={null=org.apache.solr.client.solrj.SolrServerException:Timeout
>> occured while waiting response from server at:
>> http://<machine:port>/solr},Operation splitshard caused
>> exception:=org.apache.solr.common.SolrException: SPLITSHARD failed to
>> invoke
>> SPLIT core admin command,exception={msg=SPLITSHARD failed to invoke SPLIT
>> core admin command,rspCode=500}}
>> INFO  - 2014-01-03 14:17:45.365;
>> org.apache.solr.servlet.SolrDispatchFilter;
>> [admin] webapp=null path=/admin/collections
>> params={shard=shard1&action=SPLITSHARD&collection=collection1} status=500
>> QTime=15736
>> ERROR - 2014-01-03 14:17:45.366; org.apache.solr.common.SolrException;
>> null:org.apache.solr.common.SolrException: SPLITSHARD failed to invoke
>> SPLIT
>> core admin command
>>         at
>>
>> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:200)
>>         at
>>
>> org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:422)
>>         at
>>
>> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:158)
>>         at
>>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>         at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
>>         at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
>>         at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
>>         at
>>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>>         at
>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>>         at
>>
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
>>         at
>>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>>         at
>>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>>         at
>>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>>         at
>>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
>>         at
>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>         at
>>
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>         at
>>
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>>         at
>>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>         at org.eclipse.jetty.server.Server.handle(Server.java:368)
>>         at
>>
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
>>         at
>>
>> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
>>         at
>>
>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
>>         at
>>
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
>>         at
>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
>>         at
>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>         at
>>
>> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
>>         at
>>
>> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
>>         at
>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>         at
>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>>
>> Before this exception, I am seeing couple of timeouts :
>>
>> ERROR - 2014-01-03 14:17:45.350; org.apache.solr.common.SolrException;
>> Collection splitshard of splitshard
>> failed:org.apache.solr.common.SolrException: SPLITSHARD failed to invoke
>> SPLIT core admin command
>>         at
>>
>> org.apache.solr.cloud.OverseerCollectionProcessor.collectShardResponses(OverseerCollectionProcessor.java:978)
>>         at
>>
>> org.apache.solr.cloud.OverseerCollectionProcessor.splitShard(OverseerCollectionProcessor.java:776)
>>         at
>>
>> org.apache.solr.cloud.OverseerCollectionProcessor.processMessage(OverseerCollectionProcessor.java:239)
>>         at
>>
>> org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:171)
>>         at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
>> occured
>> while waiting response from server at: http://<machine:port>/solr
>>         at
>>
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:499)
>>         at
>>
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197)
>>         at
>>
>> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
>>         at
>>
>> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118)
>>         at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>         at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>         at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>         ... 1 more
>> Caused by: java.net.SocketTimeoutException: Read timed out
>>         at java.net.SocketInputStream.socketRead0(Native Method)
>>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>         at
>>
>> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
>>         at
>>
>> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
>>         at
>>
>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
>>         at
>>
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
>>         at
>>
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
>>         at
>>
>> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
>>         at
>>
>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
>>         at
>>
>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
>>         at
>>
>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
>>         at
>>
>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
>>         at
>>
>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
>>         at
>>
>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
>>         at
>>
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
>>         at
>>
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>>         at
>>
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>>         at
>>
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>>         at
>>
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:393)
>>         ... 11 more
>>
>>
>>
>> INFO  - 2014-01-03 14:17:35.347; org.apache.solr.update.SolrIndexSplitter;
>> SolrIndexSplitter: partitions=2 segments=26
>> INFO  - 2014-01-03 14:17:38.131; org.apache.solr.update.SolrIndexSplitter;
>> SolrIndexSplitter: partition #0 range=80000000-bfffffff
>> ERROR - 2014-01-03 14:17:45.342;
>> org.apache.solr.cloud.OverseerCollectionProcessor; Error from shard:
>> <mahine:port>/solr
>> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
>> waiting response from server at: http://<machine:port>/solr
>>         at
>>
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:499)
>>         at
>>
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197)
>>         at
>>
>> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
>>         at
>>
>> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118)
>>         at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>         at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>         at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>         at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.net.SocketTimeoutException: Read timed out
>>         at java.net.SocketInputStream.socketRead0(Native Method)
>>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>         at
>>
>> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
>>         at
>>
>> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
>>         at
>>
>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
>>         at
>>
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
>>         at
>>
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
>>         at
>>
>> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
>>         at
>>
>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
>>         at
>>
>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
>>         at
>>
>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
>>         at
>>
>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
>>         at
>>
>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
>>         at
>>
>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
>>         at
>>
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
>>         at
>>
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>>
>>
>> Sorry for long stack trace.
>>
>> Though it throws these exceptions, it still creates the cores and splits
>> the
>> index based on hashrange. The UI says both the shard1 as well as shard1_0
>> and shard1_1 as active. So even though I have 2 split shards orginated
>> from
>> shard1, I am not able to delete shard1 as it is still present as active.
>>
>> I initially explored about
>> https://issues.apache.org/jira/browse/SOLR-5314
>> and https://issues.apache.org/jira/browse/SOLR-5214. But in my case
>> since it
>> looks more like timeout is triggering the issue, I upgraded from 4.5 to
>> 4.6
>> as it was mentioned in SOLR-5314 that it is fixed in 4.6. But I am still
>> getting this issue. Any help regarding this would be greatly appreciated
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/SPLITSHARD-failed-to-invoke-SPLIT-core-admin-command-tp4109431.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
> --
>
> Anshum Gupta
> http://www.anshumgupta.net
>



-- 

Anshum Gupta
http://www.anshumgupta.net

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message