nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Kosiorowski <pkosiorow...@gmail.com>
Subject Re: DistributedSearch$Client.updateSegments() blocking other threads
Date Fri, 16 Sep 2005 15:48:07 GMT
Hello Andrzej,
You can also try http://issues.apache.org/jira/browse/NUTCH-79
- I think it should also help here - it is a bit complicated as it 
contain additional functionality but if you have any problems I am 
willing to help. I am going to perform some test of it again and maybe 
commit it in some time if others think it is worth it.
Regrads
Piotr

Andrzej Bialecki wrote:
> Hi,
> 
> I was doing performance testing of a distributed search setup, with 
> JMeter, using the code from trunk/.
> 
> Whenever one of the backend Servers goes down, there is a hiccup on the 
> frontend, because all ParallelCalls started by the Client, which still 
> use that dead address, need to timeout. This is expected, and acceptable.
> 
> New calls being made in the meantime (before updateSegments() discovers 
> that the host is down) will also need to timeout - which is so so, I 
> think it could be improved by removing the offending address at the 
> first sign of trouble, i.e. not to wait for updateSegments() but 
> immediately remove the dead host from liveAddresses. Anyway, read on...
> 
> What was curious was that the same hiccup would then occur every 10 
> seconds, which is the hardcoded interval for calling 
> Client.updateSegments(). It was as if the call to updateSegments() was 
> synchronized on the whole class, so that all other calls are blocked 
> until updateSegments() completes. I modified the code, so that instead 
> of using DistributedSearch$Client itself as a Thread instance, a new 
> independent Thread instance is created.
> 
> The hiccups are gone now - the list of liveAddresses is still being 
> updated as it should whenever Servers go down/up, but now 
> updateSegments() doesn't interfere with other calls. I attach the patch 
> - but to be honest I'm still not quite sure what was happening...
> 
> 
> ------------------------------------------------------------------------
> 
> Index: DistributedSearch.java
> ===================================================================
> --- DistributedSearch.java	(revision 280515)
> +++ DistributedSearch.java	(working copy)
> @@ -112,8 +112,9 @@
>      public Client(InetSocketAddress[] addresses) throws IOException {
>        this.defaultAddresses = addresses;
>        updateSegments();
> -      setDaemon(true);
> -      start();
> +      Thread t = new Thread(this);
> +      t.setDaemon(true);
> +      t.start();
>      }
>      
>      private static final Method GET_SEGMENTS;
> @@ -168,8 +169,10 @@
>          liveSegments+=segments.length;
>        }
>  
> -      this.liveAddresses = (InetSocketAddress[]) // update liveAddresses
> -        liveAddresses.toArray(new InetSocketAddress[liveAddresses.size()]);
> +      synchronized(this.liveAddresses) {
> +        this.liveAddresses = (InetSocketAddress[]) // update liveAddresses
> +          liveAddresses.toArray(new InetSocketAddress[liveAddresses.size()]);
> +      }
>  
>        LOG.info("STATS: "+liveServers+" servers, "+liveSegments+" segments.");
>      }


Mime
View raw message