nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Kubes <nutch-...@dragonflymc.com>
Subject Single Search Server, Multiple Indexes on Separate Disks
Date Thu, 24 Aug 2006 15:23:55 GMT
We have completed our test of having a a single search server with 
multiple indexes.

Our test box has Dual Core 3.2 processor, 2G of memory and 4 small hard 
disks (60G).  Each disk has its own index of 1-2 million pages and there 
are 4 search servers running, one for each index on the box.  On a 
separate box is the search website using the search-server.txt file to 
point to the search servers on the single test box. 

With this setup we saw absolutely no slowdown in the speed at which 
search results were being returned up to about 10 queries per second.  
At 15 queries per second we are seeing both CPUs between 70-80% usage 
with load averages on the box hovering around 2.0 or a little below.  At 
20 queries per second the box goes to 98% on both CPUs and load averages 
start to spike if the traffic is constant.  Searches sometimes slow down 
because of maxing the CPUs.  Interestingly enough the slowdown has 
little to do with IO and is almost completely CPU bound.

This is in comparison to having each search server on a separate box.  
So my guess is that this type of system as it is now can handle 10 
queries a second for 8 million pages per box consistently.  We could 
probably better this result by switching to 64 bit AMD processors as 
this is CPU bound.  So in all it is about 50% less cost than a search 
server per box type of setup and would require about 1/4 to 1/3 as much 
power.

Does anybody have any ideas on how to speed this up?  Hardware, 
software, etc.  I am willing to put it into practice and test.

Dennis Kubes

Mime
View raw message