lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap
Date Fri, 11 Jan 2019 20:28:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740757#comment-16740757
] 

Michael McCandless commented on LUCENE-8635:
--------------------------------------------

Wow, this is impressive!  Surprising how small the change was – basically opening up the
FST BytesStore API a bit so that we could have an impl that wraps an {{IndexInput}} (reading
backwards) instead of a {{byte[]}}.

Can you copy/paste the rally results out of Excel here?  I'm curious what search-time impact
you're seeing.  If it not too much of an impact maybe we should consider just moving FSTs
off-heap in the default codec?  We've done similar things recently for Lucene ... e.g. moving
norms off heap.

I'll run Lucene's wikipedia benchmarks to measure the impact from our standard benchmarks
(the nightly Lucene benchmarks).

> Lazy loading Lucene FST offheap using mmap
> ------------------------------------------
>
>                 Key: LUCENE-8635
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8635
>             Project: Lucene - Core
>          Issue Type: New Feature
>          Components: core/FSTs
>         Environment: I used below setup for es_rally tests:
> single node i3.xlarge running ES 6.5
> es_rally was running on another i3.xlarge instance
>            Reporter: Ankit Jain
>            Priority: Major
>         Attachments: offheap.patch, rally_benchmark.xlsx
>
>
> Currently, FST loads all the terms into heap memory during index open. This causes frequent
JVM OOM issues if the term size gets big. A better way of doing this will be to lazily load
FST using mmap. That ensures only the required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm planning
to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be special keyword
for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based on fstOffHeap
field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using es_rally
and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message