lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ankit Jain (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap
Date Sat, 12 Jan 2019 04:59:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740855#comment-16740855
] 

Ankit Jain edited comment on LUCENE-8635 at 1/12/19 4:58 AM:
-------------------------------------------------------------

The excel sheet is big, so pasting here might not help? You have good point about moving
FSTs off-heap in the default codec as we can always preload mmap file during index open as
demonstrated [here|https://www.elastic.co/guide/en/elasticsearch/reference/master/_pre_loading_data_into_the_file_system_cache.html]

 

I ran the default lucene test suite and couple of tests seem to fail. Though, they don't
seem to have anything to do with my change:

 

   [junit4] Tests with failures [seed: 1D3ADDF6AE377902]:

   [junit4]   - org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup

   [junit4]   - org.apache.solr.cloud.autoscaling.ScheduledTriggerTest.testTrigger

   [junit4] Execution time total: 1 hour 12 minutes 40 seconds

   [junit4] Tests summary: 833 suites (7 ignored), 4024 tests, 2 failures, 286 ignored (153
assumptions)

 

UPDATE: The tests passed after retrying individually. 

 


was (Author: akjain):
The excel sheet is big, so pasting here might not help? You have good point about moving
FSTs off-heap in the default codec as we can always preload mmap file during index open as
demonstrated [here|https://www.elastic.co/guide/en/elasticsearch/reference/master/_pre_loading_data_into_the_file_system_cache.html]

 

I ran the default lucene test suite and couple of tests seem to fail. Though, they don't
seem to have anything to do with my change:

 

   [junit4] Tests with failures [seed: 1D3ADDF6AE377902]:

   [junit4]   - org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup

   [junit4]   - org.apache.solr.cloud.autoscaling.ScheduledTriggerTest.testTrigger

   [junit4]

   [junit4]

   [junit4] JVM J0:     1.40 ..  4359.18 =  4357.78s

   [junit4] JVM J1:     1.40 ..  4359.35 =  4357.95s

   [junit4] JVM J2:     1.40 ..  4359.30 =  4357.90s

   [junit4] Execution time total: 1 hour 12 minutes 40 seconds

   [junit4] Tests summary: 833 suites (7 ignored), 4024 tests, 2 failures, 286 ignored (153
assumptions)

 

Details for failing tests

 

NOTE: reproduce with: ant test  -Dtestcase=ScheduledTriggerTest -Dtests.method=testTrigger
-Dtests.seed=1D3ADDF6AE377902 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=mr-IN
-Dtests.timezone=America/St_Lucia -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

   [junit4] FAILURE 9.03s J2 | ScheduledTriggerTest.testTrigger <<<

   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<3> but was:<2>

   [junit4]    >        at __randomizedtesting.SeedInfo.seed([1D3ADDF6AE377902:7EF1EB7437F80A2F]:0)

   [junit4]    >        at org.apache.solr.cloud.autoscaling.ScheduledTriggerTest.scheduledTriggerTest(ScheduledTriggerTest.java:113)

   [junit4]    >        at org.apache.solr.cloud.autoscaling.ScheduledTriggerTest.testTrigger(ScheduledTriggerTest.java:66)

   [junit4]    >        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

   [junit4]    >        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

   [junit4]    >        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

   [junit4]    >        at java.base/java.lang.reflect.Method.invoke(Method.java:564)

   [junit4]    >        at java.base/java.lang.Thread.run(Thread.java:844)

 

NOTE: reproduce with: ant test  -Dtestcase=ScheduledMaintenanceTriggerTest -Dtests.method=testInactiveShardCleanup
-Dtests.seed=1D3ADDF6AE377902 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ha -Dtests.timezone=America/Nome
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

   [junit4] FAILURE 2.01s J0 | ScheduledMaintenanceTriggerTest.testInactiveShardCleanup
<<<

at __randomizedtesting.SeedInfo.seed([1D3ADDF6AE377902:161D84CF745E09]:0)

   [junit4]    >        at org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)

   [junit4]    >        at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:167)

   [junit4]    >        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

   [junit4]    >        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

   [junit4]    >        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

   [junit4]    >        at java.base/java.lang.reflect.Method.invoke(Method.java:564)

   [junit4]    >        at java.base/java.lang.Thread.run(Thread.java:844)

   [junit4]    > Caused by: java.util.concurrent.TimeoutException: last state: DocCollection(ScheduledMaintenanceTriggerTest_collection1//clusterstate.json/6)={

 

> Lazy loading Lucene FST offheap using mmap
> ------------------------------------------
>
>                 Key: LUCENE-8635
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8635
>             Project: Lucene - Core
>          Issue Type: New Feature
>          Components: core/FSTs
>         Environment: I used below setup for es_rally tests:
> single node i3.xlarge running ES 6.5
> es_rally was running on another i3.xlarge instance
>            Reporter: Ankit Jain
>            Priority: Major
>         Attachments: offheap.patch, rally_benchmark.xlsx
>
>
> Currently, FST loads all the terms into heap memory during index open. This causes frequent
JVM OOM issues if the term size gets big. A better way of doing this will be to lazily load
FST using mmap. That ensures only the required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm planning
to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be special keyword
for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based on fstOffHeap
field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using es_rally
and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message