jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julian Reschke (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (OAK-1941) RDB: decide on table layout
Date Tue, 02 Dec 2014 14:30:12 GMT

    [ https://issues.apache.org/jira/browse/OAK-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066382#comment-14066382
] 

Julian Reschke edited comment on OAK-1941 at 12/2/14 2:29 PM:
--------------------------------------------------------------

Minutes from a brainstorming with Marcel and Chetan:

The DocumentStore API essentially reflects the abilities of MongoDB. In particular:

1) a JSON document can be updated without reading it first (through native support for the
UpdateOp)

2) Updates can be made conditional (again through checkConditions in the UpdateOp)

3) When an update occured, MongoDB can return the previous state of the document

A relational database normally doesn't have these abilities, and this will likely cause problems
with the RDB persistence:

- concurrent updates may require retries; this is expensive and may fail if the different
nodes do not have exactly the same DB connectivity (CPU, network, ...)

- always having to fetch documents may make simple update operations more expensive

In order to address these problems, we have to try to get closer to MongoDB's capabilities.


1) Update without having the document in memory

This can be achieved by changing the data model to be one document + several updates. In a
string column, we can just modify the data format from "one JSON object" to "array of JSON
objects", where everything but the first is an UpdateOp. Upon reading, the RDBDocumentStore
would reconstruct the document. This will have a small performance penalty on read. The background
document splitting might take care of this (rewriting documents), or we may have to think
about a cleanup task.

In case where the overflow BLOB is used, we can change the datamodel to "gzipped-JSON-doc-in-BLOB"
plus "sequence-of-updates in string column".

The update operation itself might become DB-specific, for instance, the following appears
to work on Postgres:

UPDATE nodes
SET size = size + 10,
    modcount = modcount + 1,
    data = data || ', true'
    WHERE id = '0:/' and size + 10 < ...; 

2) Conditional Updates

The only conditions that are checked in practice are on the "_collisions" map, where the DocumentMK
will verify that a given revision is not in the map.

(The size of the map is currently unbounded because there's no cleanup yet)

It would be tricky to store the set of revisions separately, but we *can* introduce a counter
that increases every time the collisions map is changed. Having that, we can make updates
conditional on the collisions map being the same as a known state.

(Maintaining that information is implemented in attachment OAK-1941-cmodcount.diff)


3) Returning the previous state

Some DBs have extensions for this ("RETURNING"), some will require writing stored procedures.


[Update from Oakathon discussion in Sep 2014]:

a) We might not need the "previous" state due to potential changes in the DocumentMK. To be
discussed.
b) An alternative way to obtain the previous state would be to read the new state of the document,
locate the revision that we just wrote and return a NodeDocument reflecting the state of the
revision preceding the revision we created

Next steps:

- extract MongoDB specific test cases that simulate concurrent updates

- try to get 1) and 3) implemented on DB2, and see how this behaves for these tests

Other considerations:

- for databases where we don't want to add DB-specific code we may want to see whether a simple
improvement in the retry logic helps; right now, when retrying, we use two transactions to
read and write back; doing it in a single one with the right isolation level might already
be good enough.



was (Author: reschke):
Minutes from a brainstorming with Marcel and Chetan:

The DocumentStore API essentially reflects the abilities of MongoDB. In particular:

1) a JSON document can be updated without reading it first (through native support for the
UpdateOp)

2) Updates can be made conditional (again through checkConditions in the UpdateOp)

3) When an update occured, MongoDB can return the previous state of the document

A relational database normally doesn't have these abilities, and this will likely cause problems
with the RDB persistence:

- concurrent updates may require retries; this is expensive and may fail if the different
nodes do not have exactly the same DB connectivity (CPU, network, ...)

- always having to fetch documents may make simple update operations more expensive

In order to address these problems, we have to try to get closer to MongoDB's capabilities.


1) Update without having the document in memory

This can be achieved by changing the data model to be one document + several updates. In a
string column, we can just modify the data format from "one JSON object" to "array of JSON
objects", where evertyhing but the first is an UpdateOp. Upon reading, the RDBDocumentStore
would reconstruct the document. This will have a small performance penalty on read. The background
document splitting might take care of this (rewriting documents), or we may have to think
about a cleanup task.

In case where the overflow BLOB is used, we can change the datamodel to "gzipped-JSON-doc-in-BLOB"
plus "sequence-of-updates in string column".

The update operation itself might become DB-specific, for instance, the following appears
to work on Postgres:

UPDATE nodes
SET size = size + 10,
    modcount = modcount + 1,
    data = data || ', true'
    WHERE id = '0:/' and size + 10 < ...; 

2) Conditional Updates

The only conditions that are checked in practice are on the "_collisions" map, where the DocumentMK
will verify that a given revision is not in the map.

(The size of the map is currently unbounded because there's no cleanup yet)

It would be tricky to store the set of revisions separately, but we *can* introduce a counter
that increases everytime the collisions map is changed. Having that, we can make updates conditional
on the collisions map being the same as a known state.

(Maintaining that information is implemented in attachment OAK-1941-cmodcount.diff)


3) Returning the previous state

Some DBs have extensions for this ("RETURNING"), some will require writing stored procedures.


[Update from Oakathon discussion in Sep 2014]:

a) We might not need the "previous" state due to potential changes in the DocumentMK. To be
discussed.
b) An alternative way to obtain the previous state would be to read the new state of the document,
locate the revision that we just wrote and return a NodeDocument reflecting the state of the
revision preceding the revision we created

Next steps:

- extract MongoDB specific test cases that simulate concurrent updates

- try to get 1) and 3) implemented on DB2, and see how this behaves for these tests

Other considerations:

- for databases where we don't want to add DB-specific code we may want to see whether a simple
improvement in the retry logic helps; right now, when retrying, we use two transactions to
read and write back; doing it in a single one with the right isolation level might already
be good enough.


> RDB: decide on table layout
> ---------------------------
>
>                 Key: OAK-1941
>                 URL: https://issues.apache.org/jira/browse/OAK-1941
>             Project: Jackrabbit Oak
>          Issue Type: Sub-task
>          Components: rdbmk
>            Reporter: Julian Reschke
>            Assignee: Julian Reschke
>             Fix For: 1.2
>
>         Attachments: OAK-1941-cmodcount.diff, utf8measure.diff, with-modified-index.diff
>
>
> The current approach is to serialize the Document using JSON, and then to store either
(a) the full JSON in a VARCHAR column, or, if that column isn't wide enough, (b) to store
it in a BLOB (optionally gzipped).
> For debugging purposes, the inline VARCHAR always gets populated with the start of the
JSON serialization.
> However, with Oracle we are limited to 4000 bytes (which may be way less characters due
to non-ASCII overhead), so many document instances will use what was initially thought to
be the exception case.
> Questions:
> 1) Do we stick with JSON or do we attempt a different serialization? It might make sense
both wrt to length and performance. There might be also some code to borrow from the off-heap
serialization code.
> 2) Do we get rid of the "dual" strategy, and just always use the BLOB? The indirection
might make things more expensive, but then the total column width would drop considerably.
-- How can we do good benchmarks on this?
> (This all assumes that we stick with a model where all code is the same between database
types, except for the DDL statements; of course it's also conceivable add more vendor-specific
special cases into the Java code)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message