From oak-issues-return-10674-apmail-jackrabbit-oak-issues-archive=jackrabbit.apache.org@jackrabbit.apache.org Tue Jun 24 18:44:26 2014 Return-Path: X-Original-To: apmail-jackrabbit-oak-issues-archive@minotaur.apache.org Delivered-To: apmail-jackrabbit-oak-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5197311EFC for ; Tue, 24 Jun 2014 18:44:26 +0000 (UTC) Received: (qmail 90955 invoked by uid 500); 24 Jun 2014 18:44:26 -0000 Delivered-To: apmail-jackrabbit-oak-issues-archive@jackrabbit.apache.org Received: (qmail 90926 invoked by uid 500); 24 Jun 2014 18:44:26 -0000 Mailing-List: contact oak-issues-help@jackrabbit.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: oak-dev@jackrabbit.apache.org Delivered-To: mailing list oak-issues@jackrabbit.apache.org Received: (qmail 90906 invoked by uid 99); 24 Jun 2014 18:44:26 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Jun 2014 18:44:26 +0000 Date: Tue, 24 Jun 2014 18:44:26 +0000 (UTC) From: =?utf-8?Q?Michael_D=C3=BCrig_=28JIRA=29?= To: oak-issues@jackrabbit.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (OAK-1890) Concurrent System Login: slowdown for hight concurrenty levels MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/OAK-1890?page=3Dcom.atlassian.j= ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D140424= 97#comment-14042497 ]=20 Michael D=C3=BCrig edited comment on OAK-1890 at 6/24/14 6:44 PM: ------------------------------------------------------------- The reason for this is that cancelled {{RegistrationCallable}} tasks are no= t automatically purged from the {{ScheduledExecutorService}}'s queue but st= ay there until their time is up. With millions of logins (like in this test= ) we end up with millions of instances of {{RegistrationCallable}} and its = reachability graph, which causes the memory pressure.=20 {{ThreadPoolExecutor#purge}} might be helpful for fixing this. Will investi= gate. was (Author: mduerig): The reason for this is that cancelled {{RegistrationCallable}} tasks are no= t automatically purged from the {{ScheduledExecutorService}}'s queue but st= ay there until their time is up. With millions of logins (like in this test= ) we end up with millions of instances of {{RegistrationCallable}} and its = reachability graph, which causes the memory pressure.=20 > Concurrent System Login: slowdown for hight concurrenty levels > -------------------------------------------------------------- > > Key: OAK-1890 > URL: https://issues.apache.org/jira/browse/OAK-1890 > Project: Jackrabbit Oak > Issue Type: Bug > Components: jcr > Reporter: angela > > output of running the system login/logout test with profiling enabled: > {quote} > $ java -Dprofile=3Dtrue -Xmx2048M org.apache.jackrabbit.oak.run.Main benc= hmark LoginSystemTest Oak-Tar --concurrency 1,2,4,8,10,15,20,50 > Apache Jackrabbit Oak 1.1-SNAPSHOT > # LoginSystemTest C min 10% 50% 90% = max N > Oak-Tar 1 12 13 19 24 = 42 266 > Oak-Tar 2 12 15 20 24 = 32 496 > Oak-Tar 4 20 23 30 37 = 60 660 > Oak-Tar 8 41 67 75 85 = 95 532 > Oak-Tar 10 77 90 96 113 5= 166 122 > Oak-Tar 15 109 127 5559 5673 5= 701 27 > Oak-Tar 20 5868 5874 5928 5943 5= 944 20 > Oak-Tar 50 22116 22133 22151 22157 22= 162 50 > Profiler: top 5 stack trace(s) of 70414 ms: > 1865/21120 (8%): > at org.apache.jackrabbit.stats.RepositoryStatisticsImpl.getOrCreateRecord= er(RepositoryStatisticsImpl.java:99) > at org.apache.jackrabbit.stats.RepositoryStatisticsImpl.getCounter(Reposi= toryStatisticsImpl.java:80) > at org.apache.jackrabbit.oak.stats.StatisticManager.getCounter(StatisticM= anager.java:81) > at org.apache.jackrabbit.oak.jcr.session.SessionContext.getCounter(Sessio= nContext.java:182) > at org.apache.jackrabbit.oak.jcr.session.SessionImpl.(SessionImpl.j= ava:89) > at org.apache.jackrabbit.oak.jcr.session.SessionContext.createSession(Ses= sionContext.java:161) > at org.apache.jackrabbit.oak.jcr.session.SessionContext.getSession(Sessio= nContext.java:141) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(Reposito= ryImpl.java:260) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(Reposito= ryImpl.java:195) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest$1.run(LoginSystemT= est.java:54) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest$1.run(LoginSystemT= est.java:51) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:515) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.runTest(LoginSyste= mTest.java:51) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:279) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.execute(LoginSyste= mTest.java:33) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:288) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.access$000(AbstractTe= st.java:42) > at org.apache.jackrabbit.oak.benchmark.AbstractTest$Executor.run(Abstract= Test.java:215) > 1704/21120 (8%): > at java.lang.Throwable.fillInStackTrace(Native Method) > at java.lang.Throwable.(Throwable.java:196) > at java.lang.Exception.(Exception.java:41) > at org.apache.jackrabbit.oak.jcr.session.SessionStats.(SessionStats= .java:40) > at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.(SessionD= elegate.java:154) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl$1.(Repos= itoryImpl.java:271) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.createSessionD= elegate(RepositoryImpl.java:269) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(Reposito= ryImpl.java:255) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(Reposito= ryImpl.java:195) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest$1.run(LoginSystemT= est.java:54) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest$1.run(LoginSystemT= est.java:51) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:515) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.runTest(LoginSyste= mTest.java:51) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:279) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.execute(LoginSyste= mTest.java:33) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:288) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.access$000(AbstractTe= st.java:42) > at org.apache.jackrabbit.oak.benchmark.AbstractTest$Executor.run(Abstract= Test.java:215) > 1167/21120 (5%): > at java.lang.Throwable.fillInStackTrace(Native Method) > at java.lang.Throwable.(Throwable.java:181) > at java.lang.Exception.(Exception.java:29) > at java.lang.RuntimeException.(RuntimeException.java:32) > at java.lang.IllegalStateException.(IllegalStateException.java:27) > at java.util.concurrent.CancellationException.(CancellationExceptio= n.java:24) > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:220) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFutur= e.java:63) > at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(= Uninterruptibles.java:135) > at com.google.common.util.concurrent.Futures$4.run(Futures.java:1158) > at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorServ= ice.execute(MoreExecutors.java:297) > at com.google.common.util.concurrent.ExecutionList.executeListener(Execut= ionList.java:156) > at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java= :101) > at com.google.common.util.concurrent.ListenableFutureTask.addListener(Lis= tenableFutureTask.java:83) > at com.google.common.util.concurrent.ForwardingListenableFuture.addListen= er(ForwardingListenableFuture.java:47) > at com.google.common.util.concurrent.Futures.addCallback(Futures.java:117= 2) > at com.google.common.util.concurrent.Futures.addCallback(Futures.java:110= 8) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl$1.logout(Repos= itoryImpl.java:282) > at org.apache.jackrabbit.oak.jcr.session.SessionImpl$10.perform(SessionIm= pl.java:464) > at org.apache.jackrabbit.oak.jcr.session.SessionImpl$10.perform(SessionIm= pl.java:460) > at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(Session= Delegate.java:236) > at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.safePerform(Ses= sionDelegate.java:282) > at org.apache.jackrabbit.oak.jcr.session.SessionImpl.safePerform(SessionI= mpl.java:132) > at org.apache.jackrabbit.oak.jcr.session.SessionImpl.logout(SessionImpl.j= ava:460) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.runTest(LoginSyste= mTest.java:51) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:279) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.execute(LoginSyste= mTest.java:33) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:288) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.access$000(AbstractTe= st.java:42) > at org.apache.jackrabbit.oak.benchmark.AbstractTest$Executor.run(Abstract= Test.java:215) > 1161/21120 (5%): > at org.apache.jackrabbit.stats.RepositoryStatisticsImpl.getOrCreateRecord= er(RepositoryStatisticsImpl.java:99) > at org.apache.jackrabbit.stats.RepositoryStatisticsImpl.getCounter(Reposi= toryStatisticsImpl.java:80) > at org.apache.jackrabbit.oak.stats.StatisticManager.getCounter(StatisticM= anager.java:81) > at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.(SessionD= elegate.java:158) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl$1.(Repos= itoryImpl.java:271) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.createSessionD= elegate(RepositoryImpl.java:269) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(Reposito= ryImpl.java:255) > at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(Reposito= ryImpl.java:195) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest$1.run(LoginSystemT= est.java:54) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest$1.run(LoginSystemT= est.java:51) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:515) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.runTest(LoginSyste= mTest.java:51) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:279) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.execute(LoginSyste= mTest.java:33) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:288) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.access$000(AbstractTe= st.java:42) > at org.apache.jackrabbit.oak.benchmark.AbstractTest$Executor.run(Abstract= Test.java:215) > 1156/21120 (5%): > at java.lang.Throwable.fillInStackTrace(Native Method) > at java.lang.Throwable.(Throwable.java:196) > at java.lang.Exception.(Exception.java:41) > at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate$WarningLock.loc= k(SessionDelegate.java:695) > at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate$WarningLock.loc= k(SessionDelegate.java:700) > at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(Session= Delegate.java:213) > at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.safePerform(Ses= sionDelegate.java:282) > at org.apache.jackrabbit.oak.jcr.session.SessionImpl.safePerform(SessionI= mpl.java:132) > at org.apache.jackrabbit.oak.jcr.session.SessionImpl.logout(SessionImpl.j= ava:460) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.runTest(LoginSyste= mTest.java:51) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:279) > at org.apache.jackrabbit.oak.benchmark.LoginSystemTest.execute(LoginSyste= mTest.java:33) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.execute(AbstractTest.= java:288) > at org.apache.jackrabbit.oak.benchmark.AbstractTest.access$000(AbstractTe= st.java:42) > at org.apache.jackrabbit.oak.benchmark.AbstractTest$Executor.run(Abstract= Test.java:215) > summary: > 27%: org.apache.jackrabbit.stats > 19%: org.apache.jackrabbit.oak.jcr.session > 15%: org.apache.jackrabbit.oak.jcr.delegate > 12%: com.google.common.util.concurrent > 12%: org.apache.jackrabbit.oak.benchmark > {quote} -- This message was sent by Atlassian JIRA (v6.2#6252)