lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adrien Grand (JIRA)" <>
Subject [jira] [Commented] (LUCENE-8673) Use radix sorting when merging dimensional points
Date Thu, 31 Jan 2019 14:01:01 GMT


Adrien Grand commented on LUCENE-8673:

The title of this issue and of the PR is a bit confusing (because of the mention to "sort"),
since the whole point of this change is to stop sorting and to instead only partition around
the median value?

> Use radix sorting when merging dimensional points
> -------------------------------------------------
>                 Key: LUCENE-8673
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Ignacio Vera
>            Priority: Major
>         Attachments: Geo3D.png, LatLonPoint.png, LatLonShape.png
>          Time Spent: 50m
>  Remaining Estimate: 0h
> Following the advise of [~jpountz] in LUCENE-8623I have investigated using radix selection when
merging segments instead of sorting the data at the beginning. The results are pretty promising when
running Lucene geo benchmarks:
> {code:java}
> ||Approach||Index time (sec)||Force merge time (sec)||Index size (GB)||Reader heap (MB)||
>           ||Dev||Base||Diff ||Dev  ||Base  ||diff   ||Dev||Base||Diff||Dev||Base||Diff
> |points|241.5s|235.0s| 3%|157.2s|157.9s|-0%|0.55|0.55| 0%|1.57|1.57| 0%|
> |shapes|416.1s|650.1s|-36%|306.1s|603.2s|-49%|1.29|1.29| 0%|1.61|1.61| 0%|
> |geo3d|261.0s|360.1s|-28%|170.2s|279.9s|-39%|0.75|0.75| 0%|1.58|1.58| 0%|{code}
> In 2D the index throughput is more or less equal but for higher dimensions the impact
is quite big. In all cases the merging process requires much less disk space, I am attaching
plots showing the different behaviour and I am opening a pull request.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message