jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Dürig (JIRA) <j...@apache.org>
Subject [jira] [Comment Edited] (OAK-4122) Replace the commit semaphore in the segment node store with a commit queue
Date Mon, 11 Apr 2016 14:02:25 GMT

    [ https://issues.apache.org/jira/browse/OAK-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15223924#comment-15223924
] 

Michael Dürig edited comment on OAK-4122 at 4/11/16 2:01 PM:
-------------------------------------------------------------

A couple of benefits an explicit commit queue would give us:

* We could implement sophisticated monitoring, which would give us statistics (e.g. max, min,
mean, median 90% percentile etc.) about how long commits take, how long they stay on the queue,
what the commit rate is etc. 

* We could favour certain commits. E.g. commits from the compactor (see OAK-4015) could jump
the queue. Also checkpoint policies (see OAK-3587) could be implemented more easily though
this mechanism. 

* Instead of racing the compactor against concurrent commits and rebasing the latter later
in a number of compaction cycles (see OAK-2743), we could suspend the compactor on concurrent
commits and have it resume where it left off afterwards. This would give us finer control
over how we prioritise the compactor vs. normal commits. (OAK-3348 is a prerequisite for this
bullet point). 

* We could try to parallelise (parts of) certain commits. This probably goes along with some
POCs [~egli] has been working on lately. 

* As such queues would in most cases introduce a thread boundary we would shield the file
store from {{InterruptedException}} caused by applications threads (see OAK-2609). 

Overall I think replacing the commit semaphore with an explicit queue will open give us some
additional design space allowing us to generalise things we now have individual implementations
for. 


was (Author: mduerig):
A couple of benefits an explicit commit queue would give us:

* We could implement sophisticated monitoring, which would give us statistics (e.g. max, min,
mean, median 90% percentile etc.) about how long commits take, how long they stay on the queue,
what the commit rate is etc. 

* We could favour certain commits. E.g. commits from the compactor (see OAK-4015) could jump
the queue. Also checkpoint policies (see OAK-3587) could be implemented more easily though
this mechanism. 

* Instead of racing the compactor against concurrent commits and rebasing the latter later
in a number of compaction cycles (see OAK-2743), we could suspend the compactor on concurrent
commits and have it resume where it left off afterwards. This would give us finer control
over how we prioritise the compactor vs. normal commits. (OAK-3348 is a prerequisite for this
bullet point). 

Overall I think replacing the commit semaphore with an explicit queue will open give us some
additional design space allowing us to generalise things we now have individual implementations
for. 

> Replace the commit semaphore in the segment node store with a commit queue
> --------------------------------------------------------------------------
>
>                 Key: OAK-4122
>                 URL: https://issues.apache.org/jira/browse/OAK-4122
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: segmentmk
>            Reporter: Michael Dürig
>              Labels: performance, scalability, throughput
>             Fix For: 1.6
>
>
> {{SegmentNodeStore}} currently uses a semaphore to coordinate concurrent commits thus
relying on the scheduling algorithm of that implementation and ultimately of the JVM for in
what order commits are processed. 
> I think it would be beneficial to replace that semaphore with an explicit queue of pending
commit. This would allow us to implement a proper scheduler optimising for e.g. minimal system
load, maximal throughput or minimal latency etc. A scheduler could e.g. give precedence to
big commits and order commits along the order of its base revisions, which would decrease
the amount of work to be done in rebasing. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message