jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dirk Rudolph (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (OAK-5995) Lucene indexing with copyonread/write holding unexpectedly much files open
Date Wed, 29 Mar 2017 12:31:41 GMT

    [ https://issues.apache.org/jira/browse/OAK-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947030#comment-15947030
] 

Dirk Rudolph commented on OAK-5995:
-----------------------------------

Thanks. I'm not sure about the frequency with which the system is writing to the index. Anyway,
I got feedback form the operations team. We have quite a couple of exceptions including the
IndexCopier:
{code}
grep -Hirn "IndexCopier" logs/ | wc -l
grep: logs/._funionfs_control~: Permission denied
825370
{code}
 
 {code}
28.03.2017 13:20:22.381 *WARN* [172.19.48.185 [1490700022254] GET /libs/granite/ui/references/clientlibs/coral/references.css
HTTP/1.1] org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier [/oak:index/ntBaseLucene]
Found local copy for _2.si in MMapDirectory@/appl/aem/tmp/index/9322909280ba43419b97546267900f301b5258987a41f4d535a3489a5ee602a7/2
lockFactory=NativeFSLockFactory@/appl/aem/tmp/index/9322909280ba43419b97546267900f301b5258987a41f4d535a3489a5ee602a7/2
but size of local 237 differs from remote 0. Content would be read from remote file only
28.03.2017 13:20:22.383 *WARN* [172.19.48.185 [1490700022254] GET /libs/granite/ui/references/clientlibs/coral/references.css
HTTP/1.1] org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier [/oak:index/ntBaseLucene]
Found local copy for _2.cfe in MMapDirectory@/appl/aem/tmp/index/9322909280ba43419b97546267900f301b5258987a41f4d535a3489a5ee602a7/2
lockFactory=NativeFSLockFactory@/appl/aem/tmp/index/9322909280ba43419b97546267900f301b5258987a41f4d535a3489a5ee602a7/2
but size of local 258 differs from remote 0. Content would be read from remote file only
28.03.2017 13:20:22.386 *ERROR* [172.19.48.185 [1490700022254] GET /libs/granite/ui/references/clientlibs/coral/references.css
HTTP/1.1] org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker Could not access the
Lucene index at /oak:index/ntBaseLucene
java.io.FileNotFoundException: [tags(/oak:index/ntBaseLucene)] _2.si
         at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory.openInput(OakDirectory.java:180)
         at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.openInput(IndexCopier.java:355)
         at org.apache.lucene.codecs.lucene46.Lucene46SegmentInfoReader.read(Lucene46SegmentInfoReader.java:49)
         at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:340)
         at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:56)
         at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843)
         at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
         at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:66)
         at org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.<init>(IndexNode.java:105)
         at org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
         at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
         at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
         at org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:250)
         at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:1016)
         at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:949)
         at org.apache.jackrabbit.oak.query.ast.SelectorImpl.prepare(SelectorImpl.java:288)
         at org.apache.jackrabbit.oak.query.QueryImpl.prepare(QueryImpl.java:631)
         at org.apache.jackrabbit.oak.query.QueryEngineImpl.prepareAndSelect(QueryEngineImpl.java:298)
         at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:273)
         at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:233)
         at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:314)
         at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:308)
         at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:304)
         at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.getTree(IdentifierManager.java:133)
         at org.apache.jackrabbit.oak.security.user.AuthorizableBaseProvider.getByContentID(AuthorizableBaseProvider.java:56)
         at org.apache.jackrabbit.oak.security.user.AuthorizableBaseProvider.getByID(AuthorizableBaseProvider.java:51)
         at org.apache.jackrabbit.oak.security.user.UserProvider.getAuthorizable(UserProvider.java:211)
         at org.apache.jackrabbit.oak.security.user.UserPrincipalProvider.getPrincipals(UserPrincipalProvider.java:134)
         at org.apache.jackrabbit.oak.spi.security.principal.CompositePrincipalProvider.getPrincipals(CompositePrincipalProvider.java:71)
         at org.apache.jackrabbit.oak.spi.security.authentication.AbstractLoginModule.getPrincipals(AbstractLoginModule.java:493)
         at org.apache.jackrabbit.oak.security.authentication.user.LoginModuleImpl.commit(LoginModuleImpl.java:158)
         at org.apache.felix.jaas.boot.ProxyLoginModule.commit(ProxyLoginModule.java:57)
         at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
         at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
         at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
         at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
         at javax.security.auth.login.LoginContext.login(LoginContext.java:588)
         at org.apache.jackrabbit.oak.core.ContentRepositoryImpl.login(ContentRepositoryImpl.java:165)
         at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(RepositoryImpl.java:280)
         at com.adobe.granite.repository.impl.CRX3RepositoryImpl.login(CRX3RepositoryImpl.java:94)
         at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(RepositoryImpl.java:219)
         at org.apache.jackrabbit.oak.jcr.session.SessionImpl.impersonate(SessionImpl.java:274)
         at org.apache.sling.jcr.base.AbstractSlingRepository2.createServiceSession(AbstractSlingRepository2.java:182)
         at com.adobe.granite.repository.impl.SlingRepositoryImpl.createServiceSession(SlingRepositoryImpl.java:124)
         at org.apache.sling.jcr.base.AbstractSlingRepository2.loginService(AbstractSlingRepository2.java:331)
         at com.adobe.granite.ui.clientlibs.impl.HtmlLibraryManagerImpl.getServiceSession(HtmlLibraryManagerImpl.java:452)
         at com.adobe.granite.ui.clientlibs.impl.HtmlLibraryManagerImpl.send(HtmlLibraryManagerImpl.java:1714)
         at com.adobe.granite.ui.clientlibs.impl.HtmlLibraryManagerImpl.send(HtmlLibraryManagerImpl.java:1673)
         at com.adobe.granite.ui.clientlibs.impl.HtmlLibraryImpl.send(HtmlLibraryImpl.java:170)
         at com.adobe.granite.ui.clientlibs.impl.HtmlLibraryServlet.doGet(HtmlLibraryServlet.java:92)
         at org.apache.sling.api.servlets.SlingSafeMethodsServlet.mayService(SlingSafeMethodsServlet.java:269)
         at org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:345)
         at org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:376)
         at org.apache.sling.engine.impl.request.RequestData.service(RequestData.java:546)
         at org.apache.sling.engine.impl.filter.SlingComponentFilterChain.render(SlingComponentFilterChain.java:44)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:77)
         at com.day.cq.personalization.impl.TargetComponentFilter.doFilter(TargetComponentFilter.java:96)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.wcm.core.impl.WCMDebugFilter.doFilter(WCMDebugFilter.java:133)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.wcm.core.impl.WCMComponentFilter.filterRootInclude(WCMComponentFilter.java:362)
         at com.day.cq.wcm.core.impl.WCMComponentFilter.doFilter(WCMComponentFilter.java:177)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at org.apache.sling.engine.impl.SlingRequestProcessorImpl.processComponent(SlingRequestProcessorImpl.java:282)
         at org.apache.sling.engine.impl.filter.RequestSlingFilterChain.render(RequestSlingFilterChain.java:49)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:77)
         at com.adobe.cq.social.ugcbase.security.impl.SaferSlingPostServlet.doFilter(SaferSlingPostServlet.java:114)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.dam.core.impl.assetlinkshare.AdhocAssetShareAuthHandler.doFilter(AdhocAssetShareAuthHandler.java:389)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.dam.core.impl.servlet.ActivityRecordHandler.doFilter(ActivityRecordHandler.java:155)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:73)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:73)
         at com.adobe.granite.requests.logging.impl.RequestLoggerImpl.doFilter(RequestLoggerImpl.java:114)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.adobe.granite.csrf.impl.CSRFFilter.doFilter(CSRFFilter.java:220)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at org.apache.sling.security.impl.ContentDispositionFilter.doFilter(ContentDispositionFilter.java:181)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.wcm.core.impl.AuthoringUIModeServiceImpl.doFilter(AuthoringUIModeServiceImpl.java:367)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.wcm.core.impl.warp.TimeWarpFilter.doFilter(TimeWarpFilter.java:106)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.wcm.mobile.core.impl.redirect.RedirectFilter.doFilter(RedirectFilter.java:248)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at ....doFilter(....java:35)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter.doFilter(RequestProgressTrackerLogFilter.java:95)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.analytics.provisioning.impl.UserAuthenticationRequestFilter.doFilter(UserAuthenticationRequestFilter.java:119)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.adobe.cq.social.commons.cors.CORSAuthenticationFilter.doFilter(CORSAuthenticationFilter.java:91)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.wcm.foundation.forms.impl.FormsHandlingServlet.doFilter(FormsHandlingServlet.java:285)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.adobe.granite.optout.impl.OptOutFilter.doFilter(OptOutFilter.java:74)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at ....doFilter(....java:256)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.adobe.cq.history.impl.HistoryRequestFilter.doFilter(HistoryRequestFilter.java:107)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.day.cq.wcm.core.impl.WCMRequestFilter.doFilter(WCMRequestFilter.java:90)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.adobe.cq.mcm.campaign.servlets.CampaignCopyTracker.doFilter(CampaignCopyTracker.java:100)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at org.apache.sling.rewriter.impl.RewriterFilter.doFilter(RewriterFilter.java:83)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.adobe.granite.httpcache.impl.InnerCacheFilter.doFilter(InnerCacheFilter.java:77)
         at com.adobe.granite.httpcache.impl.InnerCacheFilter.doFilter(InnerCacheFilter.java:56)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at com.adobe.granite.rest.impl.servlet.ApiResourceFilter.doFilter(ApiResourceFilter.java:67)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at org.apache.sling.bgservlets.impl.BackgroundServletStarterFilter.doFilter(BackgroundServletStarterFilter.java:169)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at ....doFilter(....java:45)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:73)
         at com.adobe.granite.resourceresolverhelper.impl.ResourceResolverHelperImpl.doFilter(ResourceResolverHelperImpl.java:84)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:129)
         at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:68)
         at org.apache.sling.engine.impl.SlingRequestProcessorImpl.doProcessRequest(SlingRequestProcessorImpl.java:151)
         at org.apache.sling.engine.impl.SlingMainServlet.service(SlingMainServlet.java:216)
         at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:85)
         at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:79)
         at com.adobe.granite.license.impl.LicenseCheckFilter.doFilter(LicenseCheckFilter.java:308)
         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
         at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
         at org.apache.felix.http.sslfilter.internal.SslFilter.doFilter(SslFilter.java:89)
         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
         at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
         at org.apache.sling.security.impl.ReferrerFilter.doFilter(ReferrerFilter.java:290)
         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
         at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
         at org.apache.sling.featureflags.impl.FeatureManager.doFilter(FeatureManager.java:116)
         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
         at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
         at org.apache.sling.engine.impl.log.RequestLoggerFilter.doFilter(RequestLoggerFilter.java:75)
         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
         at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
         at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:129)
         at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
         at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
         at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:124)
         at org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:61)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
         at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
         at org.eclipse.jetty.server.Server.handle(Server.java:499)
         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
         at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
         at java.lang.Thread.run(Thread.java:745)
{code}

> Lucene indexing with copyonread/write holding unexpectedly much files open
> --------------------------------------------------------------------------
>
>                 Key: OAK-5995
>                 URL: https://issues.apache.org/jira/browse/OAK-5995
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: lucene
>    Affects Versions: 1.4.1
>            Reporter: Dirk Rudolph
>            Assignee: Chetan Mehrotra
>         Attachments: lsofout2.txt
>
>
> We recently faced the issue that our Oak based enterprise content management system run
into failures due to too much open files. Monitoring the lsof output we found out that most
of the opened files of the process are the files within the configured localIndexDir of the
LuceneIndexProviderService. 
> {code}
> enableCopyOnReadSupport="true"
> localIndexDir="tmp/index"
> enableCopyOnWriteSupport="true"
> {code}
> See attached the lsof output:
> {code}
> ~ wc -l lsofout2.txt
>    20388 lsofout2.txt
> ~ grep "tmp/index" lsofout2.txt | wc -l
>    13499
> {code}
> where more then 60% of open files are "tmp/index" ones as configured as {{localIndexDir}}
shortly after a restart of the process.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message