lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Harwood <>
Subject Re: Notes on distributed searching with Lucene
Date Tue, 26 Mar 2002 10:43:03 GMT
Good to see that this has promoted some discussion :)

Here is some feedback on some of the questions this has raised:

1) My application is able to parition the indexes according to some 
application-specific data. Each application would have to have its own scheme 
for partitioning.
2) The stubs/skeletons described in my sequence diagram are not the actual RMI 
stubs/skeletons used (bad naming on my part) but are essential adapters 
required to make up for some limitations in the current lucene classes.

Since implementing the initial design I am now more interested in genericising 
it and making use of mobile Java code and the visitor pattern to create a very 
powerful and flexible infrastructure. The remote interface for each of the 
index servers becomes very simple:

public interface RemoteIndex extends Remote
	public IndexVisitor visit(IndexVisitor visitor) throws RemoteException;

The index Visitor interface is for serializable Java classes that are 
"invited" into each index server and given local access to the index through a 
"visit" method:

public interface IndexVisitor
	public void visit(LocalIndexInterface localIndex);

Once inside the server different classes of visitor could conceivably perform 
any of the following activities:
* Perform a query
* Gather admin stats (memory usage, index size)
* Alter server characteristics (switch to using RAMDirectory?)
* Deposit and collect named "listeners":
Each index could use the "observer" pattern to allow visitors to plug their 
choice of listener classes into the core engine. These local listeners could 
perform tasks such as monitoring the search terms used in queries being 
performed ("top 10 searches"?)or monitor average response times. If the 
listeners are named they could be deposited by a visitor and subsequently 
collected by a different visitor some time later. RMI would also allow these 
listeners to report back to a remote object eg for centralised logging.

//example pseudo code interface....
public interface LocalIndexInterface
        public void close();
        public void optimize();
        public void search(....);

	public void registerSearchListener(SearchListener sl, String name);
	public SearchListener retrieveSearchListener(String name);
	public void unregisterSearchListener(SearchListener sl, String name);

	public void registerNewDocListener(NewDocListener nl, String name);
        // other listener hooks for index activity

This of course gives the power to deploy (and undeploy) new java code to use, 
change or monitor the behaviour of a whole server farm of indexes without 
stoppage, recompilation or redeployment. 
This level of abstraction offers great flexibility but would need to be 
balanced (same old story) with performance requirements that typically favour 
lower-level constructs without abstractions.

To unsubscribe, e-mail:   <>
For additional commands, e-mail: <>

View raw message