jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikas Saurabh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (OAK-3976) journal should support large(r) entries
Date Thu, 08 Dec 2016 12:20:58 GMT

    [ https://issues.apache.org/jira/browse/OAK-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732050#comment-15732050
] 

Vikas Saurabh commented on OAK-3976:
------------------------------------

[~mreutegg], here's roughly what I am thinking of doing:
{code}
diff --git a/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
b/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
index 04ecfd7..c168ff0 100644
--- a/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
+++ b/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
@@ -185,6 +185,13 @@ public final class DocumentNodeStore
     private boolean disableJournalDiff =
diff --git a/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
b/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
index 04ecfd7..c168ff0 100644
--- a/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
+++ b/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentNodeStore.java
@@ -185,6 +185,13 @@ public final class DocumentNodeStore
     private boolean disableJournalDiff =
             Boolean.getBoolean(SYS_PROP_DISABLE_JOURNAL);

+    public static final String SYS_PROP_JOURNAL_PUSH_THRESHOLD = "oak.journalPushThreshold";
+    /**
+     * Threshold for number of paths in journal entry to require a force push during commit
+     * (instead of at background write)
+     */
+    static int journalPushThreshold = 100000; //non-final to allow for tests.
+
     /**
      * The document store (might be used by multiple node stores).
      */
@@ -790,6 +797,7 @@ public final class DocumentNodeStore
                         changes.addChangeSet(getChangeSet(info));
                         // update head revision
                         newHead[0] = before.update(c.getRevision());
+                        pushJournalEntry(c.getRevision(), false);
                         setRoot(newHead[0]);
                         commitQueue.headRevisionChanged();
                         dispatcher.contentChanged(getRoot(), info);
@@ -2142,19 +2150,32 @@ public final class DocumentNodeStore
         return unsavedLastRevisions.persist(this, new UnsavedModifications.Snapshot() {
             @Override
             public void acquiring(Revision mostRecent) {
-                if (store.create(JOURNAL, singletonList(changes.asUpdateOp(mostRecent))))
{
-                    // success: start with a new document
-                    changes = newJournalEntry();
-                } else {
-                    // fail: log and keep the changes
-                    LOG.error("Failed to write to journal, accumulating changes for future
write (~" + changes.getMemory()
-                            + " bytes).");
-                }
+                pushJournalEntry(mostRecent, true);
             }
         }, backgroundOperationLock.writeLock());
     }

     //-----------------------------< internal >---------------------------------
+    private ReadWriteLock journalPushLock = new ReentrantReadWriteLock();
+    void pushJournalEntry(Revision r, boolean force) {
+        if (!force && changes.getNumChangedPaths() < journalPushThreshold) {
+            return;
+        }
+        journalPushLock.writeLock().lock();
+        try {
+            if (store.create(JOURNAL, singletonList(changes.asUpdateOp(r)))) {
+                // success: start with a new document
+                changes = newJournalEntry();
+            } else {
+                // fail: log and keep the changes
+                LOG.error("Failed to write to journal, accumulating changes for future write
(~" + changes.getMemory()
+                        + " bytes).");
+            }
+        } finally {
+            journalPushLock.writeLock().unlock();
+        }
+
+    }

     /**
      * Returns the binary size of a property value represented as a JSON or
{code}
A few things that I think were important to take care of:
* Commit (if required) should push journal entry before it sets new head
* minimize journal push critical section

This way things are currently, I'm not completely confident of:
# commit clears journal and pushes an entry with its own revision V/s bkWrite checked unsavedLastRevs
to find the latest rev
#* I'm not sure if commit's rev could be bigger that highest-last-rev. Also, does it matter?
# BkWrite can now potentially create empty journal entries. Should we avoid that?

> journal should support large(r) entries
> ---------------------------------------
>
>                 Key: OAK-3976
>                 URL: https://issues.apache.org/jira/browse/OAK-3976
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: documentmk
>    Affects Versions: 1.3.14
>            Reporter: Stefan Egli
>            Assignee: Vikas Saurabh
>             Fix For: 1.6, 1.5.16
>
>
> Journal entries are created in the background write. Normally this happens every second.
If for some reason there is a large delay between two background writes, the number of pending
changes can also accumulate. Which can result in (arbitrary) large single journal entries
(ie with large {{_c}} property).
> This can cause multiple problems down the road:
> * journal gc at this point loads 450 entries - and if some are large this can result
in a very large memory consumption during gc (which can cause severe stability problems for
the VM, if not OOM etc). This should be fixed with OAK-3001 (where we only get the id, thus
do not care how big {{_c}} is)
> * before OAK-3001 is done (which is currently scheduled after 1.4) what we can do is
reduce the delete batch size (OAK-3975)
> * background reads however also read the journal entries and even if OAK-3001/OAK-3975
are implemented the background read can still cause large memory consumption. So we need to
improve this one way or another.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message