lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (SOLR-2155) Geospatial search using geohash prefixes
Date Tue, 18 Jan 2011 22:44:44 GMT

     [ https://issues.apache.org/jira/browse/SOLR-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

David Smiley updated SOLR-2155:
-------------------------------

    Attachment: GeoHashPrefixFilter.patch

Attached is my latest patch for geohash prefix based geospatial search.  This patch is a performance-centric
update.  GeoHashes are still used, but I'm indexing a token for every geohash length per point
indexed.  So a geohash length of 9 results in 9 tokens.  This solves performance issues when
a huge number of points were matching a query.  See the below table:

||km||places/query||ms/query (LatLon)||ms/query (geohash)||
|11|	692|	3.8| 5.0|
|44|	4,043|	4.8|6.0|
|230|	57,200|	15.0|17.5|
|1800|	1,405,767|	94.0|71.0|

The LatLon is using a pair of trie doubles at a precisionStep of 8.  I tried 6 & 16 but
8 was about right.  The GeoHash length (a new configurable option) was chosen to be 9 which
has plenty of precision for most uses (I recall it's a couple meters or less; I forget). 
The queries are bounding lat-lon boxes.

What isn't in this performance table is the impact of this new algorithm on more complicated
spatial queries.  It's superior to the algorithm that existed before it and it should also
be superior to LatLonType.  Grid boxes that are completely within the query shape get efficiently
added in one fell swoop.

Code details:
* Most of the former patch, which included a lot of additions to GeoHashUtils is no longer
present in this new patch.  This was basically a rewrite.
* I abstracted use of GeoHashUtils to GridNode.GridReferenceSystem class so that in the future
I can tinker with alternate more efficient encodings without breaking any code here.
* I needed a shape interface or abstract class and so I decided to embrace & extend org.apache.lucene.spatial.geometry.shape.Geometry2D
instead of having my own like I did before.  I added PointDistanceGeom & MultiGeom.
* There is an extensive random data filter test in SpatialFilterTest that I added.  It's hard
to follow but it teased out a few bugs.

Next patch real soon:
* I'm going to modify the build.xml to grab the LGPL licensed JTS library which has well-tested
& high performance geometry code.  In particular, I'll use it to implement a polygon shape.
 (Already done in another codebase; just needs to be ported to this patch)
* I'm going to include an alternative query parser to what comes with Solr.  This one will
do all of point-distance, lat-lon box, and polygon. (Already done in another codebase; just
needs to be ported to this patch).

Future:
* Replace geohash with something more efficient.  Some basic testing suggests to me I could
double-or-better the performance.
* Compatibility with distance sorting / relevancy boosting when not multi-valued.

I'd really like input from other geospatial birds-of-a-feather in Solr, especially committers.

As an aside, MongoDB has chosen a similar algorithm.

> Geospatial search using geohash prefixes
> ----------------------------------------
>
>                 Key: SOLR-2155
>                 URL: https://issues.apache.org/jira/browse/SOLR-2155
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: David Smiley
>         Attachments: GeoHashPrefixFilter.patch, GeoHashPrefixFilter.patch
>
>
> There currently isn't a solution in Solr for doing geospatial filtering on documents
that have a variable number of points.  This scenario occurs when there is location extraction
(i.e. via a "gazateer") occurring on free text.  None, one, or many geospatial locations might
be extracted from any given document and users want to limit their search results to those
occurring in a user-specified area.
> I've implemented this by furthering the GeoHash based work in Lucene/Solr with a geohash
prefix based filter.  A geohash refers to a lat-lon box on the earth.  Each successive character
added further subdivides the box into a 4x8 (or 8x4 depending on the even/odd length of the
geohash) grid.  The first step in this scheme is figuring out which geohash grid squares cover
the user's search query.  I've added various extra methods to GeoHashUtils (and added tests)
to assist in this purpose.  The next step is an actual Lucene Filter, GeoHashPrefixFilter,
that uses these geohash prefixes in TermsEnum.seek() to skip to relevant grid squares in the
index.  Once a matching geohash grid is found, the points therein are compared against the
user's query to see if it matches.  I created an abstraction GeoShape extended by subclasses
named PointDistance... and CartesianBox.... to support different queried shapes so that the
filter need not care about these details.
> This work was presented at LuceneRevolution in Boston on October 8th.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message