lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Underlying file changed by an external force
Date Wed, 10 May 2017 22:59:26 GMT
bq: All the core.properties files are each in their own directory with
no overlap

Not quite what I was asking. By definition, all core.properties are in
their own directory. In fact Solr stops looking down the tree when it
finds the first directory with core.properties in it and immediately
moves on to the next sibling directory.

_Inside_ the core.properties files, are there any dataDir properties
pointing to the same place as in any other core.properties? Note,
dataDir properties usually aren't even present unless you did
something special so don't be surprised if there's nothing there.

Best,
Erick

On Wed, May 10, 2017 at 10:56 AM, Oakley, Craig (NIH/NLM/NCBI) [C]
<craig.oakley@nih.gov> wrote:
>> You need to look at all of your core.properties files and see if any of them point
to the same data directory.
>
> All the core.properties files are each in their own directory with no overlap.
>
>> Second: if you issue a "kill -9" you can leave write locks lingering.
>
> We manage our Solr instances with supervisor, which can send a "kill -9" if "kill -6"
does not suffice; but the problem tends to manifest itself at some time other than startup
>
> The Solr version is 5.4.1, in case that is relevant.
>
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Thursday, May 04, 2017 3:20 PM
> To: solr-user <solr-user@lucene.apache.org>
> Subject: Re: Underlying file changed by an external force
>
> You need to look at all of your core.properties files and see if any
> of them point to the same data directory.
>
> Second: if you issue a "kill -9" you can leave write locks lingering.
>
> Best,
> Erick
>
> On Thu, May 4, 2017 at 11:00 AM, Oakley, Craig (NIH/NLM/NCBI) [C]
> <craig.oakley@nih.gov> wrote:
>> We have been having problems with different collections on different SolrCloud clusters,
all seeming to be related to the write.lock file with stack traces similar to the following.
Are there any suggestions what might be the cause and what might be the solution? Thanks
>>
>>
>> org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external
force at 2017-04-13T20:43:08.630152Z, (lock=NativeFSLock(path=/data/solr/biosample/dba_test_shard1_replica1/data/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807
exclusive valid],ctime=2017-04-13T20:43:08.630152Z))
>>
>>        at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179)
>>
>>        at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:37)
>>
>>        at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:732)
>>
>>        at org.apache.lucene.index.IndexFileDeleter.deletePendingFiles(IndexFileDeleter.java:503)
>>
>>        at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:448)
>>
>>        at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2099)
>>
>>        at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2041)
>>
>>        at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1083)
>>
>>        at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1125)
>>
>>        at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:131)
>>
>>        at org.apache.solr.update.DefaultSolrCoreState.changeWriter(DefaultSolrCoreState.java:183)
>>
>>        at org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:207)
>>
>>        at org.apache.solr.core.SolrCore.reload(SolrCore.java:472)
>>
>>        at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:849)
>>
>>        at org.apache.solr.handler.admin.CoreAdminHandler.handleReloadAction(CoreAdminHandler.java:768)
>>
>>        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:230)
>>
>>        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:184)
>>
>>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>>
>>        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664)
>>
>>        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:438)
>>
>>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223)
>>
>>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
>>
>>        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>
>>        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>
>>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>
>>        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>>
>>        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>>
>>        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>
>>        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>
>>        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>
>>        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>
>>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>
>>        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>>
>>        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>>
>>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>
>>        at org.eclipse.jetty.server.Server.handle(Server.java:499)
>>
>>        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>
>>        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>
>>        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>
>>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>
>>        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>
>>        at java.lang.Thread.run(Thread.java:745)
>>

Mime
View raw message