jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chetan Mehrotra (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (OAK-5192) Reduce Lucene related growth of repository size
Date Mon, 03 Jul 2017 12:05:00 GMT

    [ https://issues.apache.org/jira/browse/OAK-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072317#comment-16072317

Chetan Mehrotra commented on OAK-5192:

Attached binSize100.txt, binSize16384.txt, binSizeTotal.txt and nonBinSizeTotal.txt with the
sizes for binaries <= 100, <=16384 and <= Long.MaxValue and the sizes for non binaries
per revision in diff.txt.zip, respectively.

Total sum of sizes are: 44785 bytes, 5887029, bytes, 237895837 bytes and 245128 bytes listed
in the same order as above

Had a look at stats again. 

$ grep ',' binSize16384.txt | cut -f 1 -d L | awk '$1<4000' | awk '{n += $1}; END{print

So checking the size of blobs < 4kb (typical inline size) via following command shows that
actual index size in segment nodestore is 380KB. 

So from number it appears that impact of Lucene growth on SegmentNodeStore with a suggested
setup involving maxInlineSize of no more than 4kb should not be very high. The earlier analysis
was done on a setup where inline size was set to 16KB which caused a higher impact.

We should still check out ways to reduce the growth in DataStore side but from SegmentNodeStore
we should be ok with current setup

/cc [~teofili] [~mduerig] [~tmueller]

> Reduce Lucene related growth of repository size
> -----------------------------------------------
>                 Key: OAK-5192
>                 URL: https://issues.apache.org/jira/browse/OAK-5192
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: lucene, segment-tar
>            Reporter: Michael Dürig
>            Assignee: Chetan Mehrotra
>              Labels: perfomance, scalability
>             Fix For: 1.8, 1.7.8
>         Attachments: added-bytes-zoom.png, binSize100.txt, binSize16384.txt, binSizeTotal.txt,
diff.txt.zip, nonBinSizeTotal.txt, OAK-5192.0.patch
> I observed Lucene indexing contributing to up to 99% of repository growth. While the
size of the index itself is well inside reasonable bounds, the overall turnover of data being
written and removed again can be as much as 99%. 
> In the case of the TarMK this negatively impacts overall system performance due to fast
growing number of tar files / segments, bad locality of reference, cache misses/thrashing
when looking up segments and vastly prolonged garbage collection cycles.

This message was sent by Atlassian JIRA

View raw message