jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chetan Mehrotra (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (OAK-5192) Reduce Lucene related growth of repository size
Date Mon, 03 Jul 2017 15:21:00 GMT

    [ https://issues.apache.org/jira/browse/OAK-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072619#comment-16072619
] 

Chetan Mehrotra commented on OAK-5192:
--------------------------------------

bq. how can I set this up in a unit test like https://github.com/tteofili/jackrabbit-oak/blob/oak-5192/oak-lucene/src/test/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneSegmentStatsTest.java
?

[~teofili] You would need to configure a FileDataStore with minRecordLength set to 4000. A
rough approach is below which you would need to adapt

{noformat}
        NodeStore nodeStore;
        try {
            fileStore = FileStoreBuilder.fileStoreBuilder(DIRECTORY)
                    .withStatisticsProvider(new DefaultStatisticsProvider(scheduledExecutorService))
                    .withBlobStore(createBlobStore())
                    .build();
            nodeStore = SegmentNodeStoreBuilders.builder(fileStore).build();
        } catch (IOException | InvalidFileStoreVersionException e) {
            throw new RuntimeException(e);
        }

    BlobStore createBlobStore(){
        FileDataStore fds = new OakFileDataStore();
        fds.setPath(new File("path/tods"));
        fds.setMinRecordLength(4000);
        fds.init(null);
        return new DataStoreBlobStore(fds);
    }

{noformat}



> Reduce Lucene related growth of repository size
> -----------------------------------------------
>
>                 Key: OAK-5192
>                 URL: https://issues.apache.org/jira/browse/OAK-5192
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: lucene, segment-tar
>            Reporter: Michael Dürig
>            Assignee: Tommaso Teofili
>              Labels: perfomance, scalability
>             Fix For: 1.8, 1.7.8
>
>         Attachments: added-bytes-zoom.png, binSize100.txt, binSize16384.txt, binSizeTotal.txt,
diff.txt.zip, nonBinSizeTotal.txt, OAK-5192.0.patch, Screen Shot 2017-07-03 at 16.50.00.png
>
>
> I observed Lucene indexing contributing to up to 99% of repository growth. While the
size of the index itself is well inside reasonable bounds, the overall turnover of data being
written and removed again can be as much as 99%. 
> In the case of the TarMK this negatively impacts overall system performance due to fast
growing number of tar files / segments, bad locality of reference, cache misses/thrashing
when looking up segments and vastly prolonged garbage collection cycles.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message