lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eks dev <>
Subject Re: ParallelMultiSearcher reimplementation
Date Mon, 13 Nov 2006 21:27:34 GMT
maybe someone interested.
I just remembered, we tested pure Hadop RPC a few (5+) months ago in simple setup, kind of
balancing server getting and distributing requests to 3 "search units"...

we went that far as java RMI proved to have ugly latency problems (or we did not get it right,
don't know for sure) in well-under-second scenario. 

All in all, spawning new thread (hadoop RPC) on java 1.5, roundtrip over 2 net nodes (100MB
) there and back was slightly under 3k / second on a few kB  message. 

----- Original Message ----
From: Doug Cutting <>
Sent: Monday, 13 November, 2006 9:50:28 PM
Subject: Re: ParallelMultiSearcher reimplementation

Chuck Williams wrote:
> I followed this same logic in ParallelWriter and got burned.  My first
> implementation (still the version submitted as a patch in jira) used
> dynamic threads to add the subdocuments to the parallel subindexes
> simultaneously.  This hit a problem with abnormal native heap OOM's in
> the jvm.  At first I thought it was simply a thread stack size / java
> heap size configuration issue, but adjusting these did not resolve the
> issue.  This was on linux.  ps -L showed large numbers of defunct
> threads.  jconsole showed enormous growing total-ever-allocated thread
> counts.  I switched to a thread pool and the issue went away with the
> same config settings.

Can you demonstrate the problem with a standalone program?

Way back in the 90's I implemented a system at Excite that spawned one 
or more Java threads per request, and it ran for days on end, handling 
20 or more requests per second.  The thread spawning overhead was 
insignificant.  That was JDK 1.2 on Solaris.  Have things gotten that 
much worse in the interim?  Today Hadoop's RPC allocates a thread per 
connection, and we see good performance.  So I certainly have 


To unsubscribe, e-mail:
For additional commands, e-mail:

Send instant messages to your online friends 

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message