jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Parvulescu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (OAK-3612) The string/template cache of the SegmentWriter should be cleared before cleanup
Date Mon, 16 Nov 2015 13:32:11 GMT

    [ https://issues.apache.org/jira/browse/OAK-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006660#comment-15006660
] 

Alex Parvulescu commented on OAK-3612:
--------------------------------------

one problem with the patch: it introduces a deadlock [1]. run _SegmentCompactionIT_ with 10
writers for a bit to verify it's not just my setup, also see the existing warning in the FileStore
about possible deadlocks when locking access to the tracker's writer [0].

I don't disagree with what you are saying, but changes like these have impacts that we cannot
predict, so I'd rather approach this with a bit more caution.

[0]
https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L752
[1]
{code}
Found one Java-level deadlock:
=============================
"RandomWriter":
  waiting to lock monitor 0x00007f89ab006518 (object 0x00000007000c73c8, a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter),
  which is held by "RandomWriter"
"RandomWriter":
  waiting for ownable synchronizer 0x00000007001311b8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "TarMK flush thread [target/SegmentCompactionIT4821636020606907980dir],
active since Mon Nov 16 14:24:48 CET 2015, previous max duration 356ms"
"TarMK flush thread [target/SegmentCompactionIT4821636020606907980dir], active since Mon Nov
16 14:24:48 CET 2015, previous max duration 356ms":
  waiting to lock monitor 0x00007f89ab006518 (object 0x00000007000c73c8, a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter),
  which is held by "RandomWriter"

Java stack information for the threads listed above:
===================================================
"RandomWriter":
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeTemplate(SegmentWriter.java:1157)
	- waiting to lock <0x00000007000c73c8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1326)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeAdded(SegmentWriter.java:1344)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:413)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1341)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1352)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:417)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1341)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1352)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:417)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1341)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1352)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:417)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1341)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:85)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:214)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:81)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:346)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:337)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.addRandomNode(SegmentCompactionIT.java:550)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.modify(SegmentCompactionIT.java:501)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.access$4(SegmentCompactionIT.java:487)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter$1.call(SegmentCompactionIT.java:466)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter$1.call(SegmentCompactionIT.java:1)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.run(SegmentCompactionIT.java:446)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:457)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:1)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
"RandomWriter":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007001311b8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.writeSegment(FileStore.java:1257)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.flush(SegmentWriter.java:304)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.prepare(SegmentWriter.java:487)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeTemplate(SegmentWriter.java:1228)
	- locked <0x00000007000c73c8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1326)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:1)
	at org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(MergingNodeStateDiff.java:73)
	at org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.childNodeChanged(MergingNodeStateDiff.java:92)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:531)
	at org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(MergingNodeStateDiff.java:70)
	at org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.childNodeChanged(MergingNodeStateDiff.java:92)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:488)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compareBranch(MapRecord.java:565)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:470)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compareBranch(MapRecord.java:565)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:470)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
	at org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(MergingNodeStateDiff.java:70)
	at org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(MergingNodeStateDiff.java:65)
	at org.apache.jackrabbit.oak.plugins.commit.ConflictHook.processCommit(ConflictHook.java:53)
	at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:484)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:507)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:563)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:226)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter$1.call(SegmentCompactionIT.java:473)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter$1.call(SegmentCompactionIT.java:1)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.run(SegmentCompactionIT.java:446)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:457)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:1)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
"TarMK flush thread [target/SegmentCompactionIT4821636020606907980dir], active since Mon Nov
16 14:24:48 CET 2015, previous max duration 356ms":
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.dropCache(SegmentWriter.java:1043)
	- waiting to lock <0x00000007000c73c8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.clearCache(SegmentTracker.java:191)
	- locked <0x000000070017fd08> (a org.apache.jackrabbit.oak.plugins.segment.SegmentTracker)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:816)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.flush(FileStore.java:776)
	- locked <0x00000007001b9c48> (a java.util.concurrent.atomic.AtomicReference)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore$1.run(FileStore.java:486)
	at java.lang.Thread.run(Thread.java:744)
	at org.apache.jackrabbit.oak.plugins.segment.file.BackgroundThread.run(BackgroundThread.java:71)

Found 1 deadlock.
{code}

> The string/template cache of the SegmentWriter should be cleared before cleanup
> -------------------------------------------------------------------------------
>
>                 Key: OAK-3612
>                 URL: https://issues.apache.org/jira/browse/OAK-3612
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segmentmk
>            Reporter: Francesco Mari
>             Fix For: 1.4
>
>         Attachments: OAK-3612-01.patch
>
>
> The string/template cache of the SegmentWriter (org.apache.jackrabbit.oak.plugins.segment.SegmentWriter#records)
is not cleared before the cleanup phase. This might maintain in-memory references to segments,
thus preventing them to be cleaned up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message