lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] [Commented] (LUCENE-5780) OrdinalMap's mapping from global ords to segment ords is sometimes wasteful
Date Fri, 20 Jun 2014 10:00:28 GMT


Robert Muir commented on LUCENE-5780:

I agree conceptually its a bug, but I think this should be 4.10. Its not anything introduced
in 4.9, its been this way since 4.2

I already am pushing back on other issues such as SOLR-6178, because i ran 125 rounds of tests
last night and want to keep things stable.

> OrdinalMap's mapping from global ords to segment ords is sometimes wasteful
> ---------------------------------------------------------------------------
>                 Key: LUCENE-5780
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Adrien Grand
>            Assignee: Adrien Grand
>             Fix For: 4.9, 5.0
> Robert found a case when the ordinal map can be quite wasteful in terms of memory usage:
in order to be able to resolve values given a global ordinals, it stores two things:
>  - an identifier of the segment where the value is
>  - the difference between the ordinal on the segment and the global ordinal
> The issue is that OrdinalMap currently picks any of the segments that contain the value
but we can do better: we can pick the first segment that has the value. This will help for
two reasons:
>  - it will potentially require fewer bits per value to store the segment ids if NRT segments
don't introduce new values
>  - if all values happen to appear in the first segment, then the map from global ords
to deltas only stores zeros.
> I just tested on an index where all values are in the first segment and this helped reduce
memory usage of the ordinal map by 4x (from 3.5MB to 800KB).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message