lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ankit Jain (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-8671) Add setting for moving FST offheap/onheap
Date Tue, 19 Feb 2019 22:16:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772384#comment-16772384
] 

Ankit Jain commented on LUCENE-8671:
------------------------------------

bq. But can you use the existing attributes instead of adding a new readerAttributes?  And
could we make this something a custom Codec impl would set?  Then we shouldn't need any changes
to FieldInfo.java, IndexWriter.java, LiveIndexWriterConfig.java, etc.  We'd just make a custom
codec setting this attribute for fields where we want to override Lucene's (BlockTreeTermReader's)
default behavior.  Yes, it'd mean one must commit at indexing time as to which fields will
be on vs off heap at search time, but I think that's an OK tradeoff?
I like this idea, just did not want it to be indexing time decision. Given performance implications
are not significant and we were discussing making offheap as default earlier, most users eventually
will have it on during indexing also.

> Add setting for moving FST offheap/onheap
> -----------------------------------------
>
>                 Key: LUCENE-8671
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8671
>             Project: Lucene - Core
>          Issue Type: New Feature
>          Components: core/FSTs, core/store
>            Reporter: Ankit Jain
>            Priority: Minor
>         Attachments: offheap_generic_settings.patch, offheap_settings.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> While LUCENE-8635, adds support for loading FST offheap using mmap, users do not have
the  flexibility to specify fields for which FST needs to be offheap. This allows users to
tune heap usage as per their workload.
> Ideal way will be to add an attribute to FieldInfo, where we have put/getAttribute. Then
FieldReader can inspect the FieldInfo and pass the appropriate On/OffHeapStore when creating
its FST. It can support special keywords like ALL/NONE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message