lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: 2.4 Performance
Date Wed, 19 Nov 2008 10:15:06 GMT

Can you describe the queries in more detail?  Can you narrow down  
exactly which operations / types of queries are substantially slower?

Also, I'm assuming both of you are NOT on Windows?  NIOFSDirectory has  
poor performance on Windows due to this bug in Sun's JVM:


Eric Bowman wrote:

> wrote:
>> On an index of around 20 gigs I've been seeing a performance drop of
>> around 35% after upgrading to 2.4 (measured on ~10000 requests
>> identical requests, executed in parallel against a threaded lucene /
>> apache setup, after a roughly 10000 query warmup). The principal
>> changes I've made so far are just to switch to NIOFSDirectories and
>> use read-only index readers.
>> Our design is roughly as follows: we have some pre-query filters,
>> queries typically involving around 25 clauses, and some
>> post-processing of hits. We collect counts and filter post query  
>> using
>> a hit collector, which uses the (now deprecated) bits() method of
>> Filters.
>> I looked at converting us to use the new DocIdSet infrastructure (to
>> gain the supposed 30% speed bump), but this seems to be somewhat
>> problematic as there is no guarantee for whether we will get back a
>> set we can do binary operations on (for example, if we get back a
>> SortedVIntList, we're pretty much out of luck - the cardinality of  
>> the set
>> is large (as it's a sortedvintlist), so we can't coerce it into
>> another type, and it doesn't have the set operations we need to use  
>> it
>> directly.
>> Has anyone else seen this? Is there anything else
>> we should be changing in the upgrade to 2.4?
> We are seeing something very similar to this.  We had an IndexReader  
> per
> core, and performance was ok.  Since upgrading to 2.4, performance has
> gone down, in some cases quite dramatically.  We switched to a single
> IndexReader, and things got better.
> We are now also looking at NIOFSDirectory after noticing a lot of
> contention in IndexReader, but that didn't help as much as we had  
> hoped.
> To be honest I'm considering trying to downgrade back to 2.3 for now,
> this is looking like a step backwards for us.
> Thanks for any suggestions,
> Eric
> -- 
> Eric Bowman
> Boboco Ltd
> +35318394189/+353872801532
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message