lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: Extending scoring to eliminate sorting on timestamp
Date Fri, 26 Jan 2007 19:15:05 GMT

: I used String because the timestamp is a Long and there wasn't any
: SortField.LONG (I guess I should have used SortField.CUSTOM). In this
: case, what should the indexing call look like? Currently, I have:

:     doc.add(new Field("timestamp",Long.toString(timestamp),Field.Store.NO,Field.Index.UN_TOKENIZED));

for the record, what you've got there won't sort correctly as a string
anyway, the number 923456 will sort after the number 12345678 because as a
string it's lexigraphicaly larger ... you'd need to 0 pad the value, or
use something like NumberTools.

i never really noticed there was an int field cache, but no long field
cache ... either way, if you are tryingto avoid hte FieldCachebecause of
hte time ittakes to initialize and not hte memory it takes up, then long
support in FieldCachewouldnt' help you.

: The other thing I was considering is to automatically limit the number
: of results (there is no way a user can grok 3 million results anyway) by
: breaking down the range filter into a series of range filters and
: executing multiple searches in series until the max number of results

i'm not sure how that would help you unlessy ou did all of the sorting
external to Lucene (or used something like the patch i mentioned)

: case (when the number of results is reasonable). One way around this is
: to execute an initial search just to figure out the number of hits
: (without sorting, without scoring) and then apply different strategies,

if you try this use Sort.INDEXORDER .. short of writing your own
HitCollector i believe it's the best way to eliminate any extra work
Lucene might do for you to sort results.

: The patch you pointed out looks very very promising.

There's been some disagreement as to how usefull it is -- i don't know if
anyone has had time to do a thorough performnce analysys of it in
differnet sitautions.  if you try it out please comment in the issue with
your experiences 9good or bad)


----- Original Message ----
From: Chris Hostetter <hossman_lucene@fucit.org>
To: java-user@lucene.apache.org
Sent: Thursday, January 25, 2007 9:09:12 PM
Subject: Re: Extending scoring to eliminate sorting on timestamp


: For various reasons, we'd like to eliminate the sort step.

can you elaborate on what those reasons are?

FunctionQuery (in the solr code base, you'll find lots of discussing in
the archives of this list) can let you use a numeric field value in the
score calculation, but it still uses the FieldCache so if you are trying
to avoid that for space/time reasons it won't help.

you may also be interested in this patch...

  https://issues.apache.org/jira/browse/LUCENE-769

in the general case it should be slower then standard sorting, but if
you are dealing with an extremely large index and your result sets all
tend to be small, it may be faster (and it won't pay the initial
FieldCache setup time on frequently modified indexes)

:                    new SortField("timestamp",SortField.STRING,true)}));

why are you sorting timestamps as strings? ... if you sort them as ints,
your FieldCache will be a whole hell of a lot smaller (i'm guessing very
few documents have identicle timestamps, so your FieldCache should be at
least half as big if you sort on ints (and probably a lot more).


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org




:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message