jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Dürig (JIRA) <j...@apache.org>
Subject [jira] [Updated] (OAK-2587) observation processing too eager/unfair under load
Date Fri, 06 Mar 2015 21:34:38 GMT

     [ https://issues.apache.org/jira/browse/OAK-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Michael Dürig updated OAK-2587:
    Attachment: OAK-2587.patch

Tentative patch. Instead of eagerly polling through the queue each iteration is set as completion
action thus delegating it back to the executor. With this I see calls to {{EventListener.onEvent}}
taking place on alternating threads even when there is a backlog to process. Basic unit tests
pass. Needs more testing beyond that though. 

> observation processing too eager/unfair under load
> --------------------------------------------------
>                 Key: OAK-2587
>                 URL: https://issues.apache.org/jira/browse/OAK-2587
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 1.0.12
>            Reporter: Stefan Egli
>            Priority: Critical
>         Attachments: OAK-2587.patch
> The current implementation of oak's observation event processing is too eager and thus
unfair under load scenarios. 
> Consider having many (eg 200) Eventlisteners but only a relatively small threadpool (eg
5 as is the default in sling) backing them. When processing changes for a particular BackgroundObserver,
that one (in BackgroundObserver.completionHandler.call) currently processes *all changes irrespective
of how many there are* - ie it is *eager*. Only once that BackgroundObserver processed all
changes will it let go and 'pass the thread' to the next BackgroundObserver. Now if for some
reason changes (ie commits) are coming in while a BackgroundObserver is busy processing an
earlier change, this will lengthen that while loop. As a result the remaining (eg 195) *EventListeners
will have to wait for a potentially long time* until it's their turn - thus *unfair*.
> Now combine the above pattern with a scenario where mongo is used as the underlying store.
In that case in order to remain highly performant it is important that the diffs (for compareAgainstBaseState)
are served from the MongoDiffCache for as many cases as possible to avoid doing a round-trip
to mongoD. The unfairness in the BackgroundObservers can now result in a large delay between
the 'first' observers getting the event and the 'last' one (of those 200). When this delay
increases due to a burst in the load, there is a risk of the diffs to no longer be in the
cache - those last observers are basically kicked out of the (diff) cache. Once this happens,
*the situation gets even worse*, since now you have yet new commits coming in and old changes
still having to be processed - all of which are being processed through in 'stripes of 5 listeners'
before the next one gets a chance. This at some point results in a totally inefficient cache
behavior, or in other words, at some point all diffs have to be read from mongoD.
> To avoid this there are probably a number of options - a few one that come to mind:
> * increase thread-pool to match or be closer to the number of listeners (but this has
other disadvantages, eg cost of thread-switching)
> * make BackgroundObservers fairer by limiting the number of changes they process before
they give others a chance to be served by the pool.

This message was sent by Atlassian JIRA

View raw message