lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-8671) Add setting for moving FST offheap/onheap
Date Tue, 19 Feb 2019 16:15:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772068#comment-16772068
] 

Michael McCandless commented on LUCENE-8671:
--------------------------------------------

Actually I think this is a good use case for the existing attributes in {{FieldInfo}} –
this sort of extensibility is exactly why we have attributes.

But can you use the existing {{attributes}} instead of adding a new {{readerAttributes}}? 
And could we make this something a custom {{Codec}} impl would set?  Then we shouldn't need
any changes to {{FieldInfo.java}}, {{IndexWriter.java}}, {{LiveIndexWriterConfig.java}}, etc. 
We'd just make a custom codec setting this attribute for fields where we want to override
Lucene's ({{BlockTreeTermReader}}'s) default behavior.  Yes, it'd mean one must commit at
indexing time as to which fields will be on vs off heap at search time, but I think that's
an OK tradeoff?

> Add setting for moving FST offheap/onheap
> -----------------------------------------
>
>                 Key: LUCENE-8671
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8671
>             Project: Lucene - Core
>          Issue Type: New Feature
>          Components: core/FSTs, core/store
>            Reporter: Ankit Jain
>            Priority: Minor
>         Attachments: offheap_generic_settings.patch, offheap_settings.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> While LUCENE-8635, adds support for loading FST offheap using mmap, users do not have
the  flexibility to specify fields for which FST needs to be offheap. This allows users to
tune heap usage as per their workload.
> Ideal way will be to add an attribute to FieldInfo, where we have put/getAttribute. Then
FieldReader can inspect the FieldInfo and pass the appropriate On/OffHeapStore when creating
its FST. It can support special keywords like ALL/NONE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message