lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Elschot <>
Subject Re: "Advanced" query language
Date Sat, 17 Dec 2005 22:36:41 GMT

While maintaining my bookmarks I ran into this:
"Case Study: Enabling Low-Cost XML-Aware Searching
Capable of Complex Querying":

Some loose thoughts:

In the system described there a Lucene document is used for each
low level xml construct, even when it contains very few characters of text.
The resulting Lucene indexes are at least 2.5 times the size of the
original document, which is not a surprise given this document structure.
Normal index size is about one third of  the indexed text.

I don't know about the XQuery standard, but I was wondering
whether this unusual document structure and the non straightforward
fit between Lucene queries and XQuery queries are related.

As for the  joines and iterations over items from the stream of XML
results: iteration over matching XML constructs should be no problem
in Lucene. Joins in Lucene are normally done via boolean filters,
so I was wondering how XQuery joins fit these.
The case study above has a note a the end of par 5.3: 
"The Search Result list that comes back could then be organized
by document id to group together all the results for a single XML
document. This is not provided by default, but has been done with
extension to this code."

Paul Elschot

On Friday 16 December 2005 03:45, Wolfgang Hoschek wrote:
> I think implementing an XQuery Full-Text engine is far beyond the  
> scope of Lucene.
> Implementing a building block for the fulltext aspect of it would be  
> more manageable. Unfortunately The W3C fulltext drafts  
> indiscriminately mix and mingle two completely different languages  
> into a single language, without clear boundaries. That's why most  
> practical folks implement XQuery fulltext search via extension  
> functions rather than within XQuery itself. This also allows for much  
> more detailed tokenization, configuration and extensibility than what  
> would be possible with the W3C draft.
> Wolfgang.
> On Dec 15, 2005, at 4:20 PM, JOAQUIN.DELGADO@ORACLE.COM wrote:
> > Mark,
> >
> > This is very cool. When I was at TripleHop we did something very  
> > similar where both query and results conformed to an XML Schema and  
> > we used XML over HTTP as our main vehicle to do remote/federated  
> > searches with quick rendering with stylesheets.
> >
> > That however is the first piece of the puzzle. If you really want  
> > to go beyond search (in the traditional sense) and be able to  
> > perform more complex operations such as joines and iterations over  
> > items from the stream of XML results you are getting you should  
> > consider implementing an XQuery Full-Text engine with Lucene  
> > adopting the now standard XQuery language.
> >
> > Here is the pointer to the working draft on the W3C working draft  
> > on XQuery 1.0 and XPath 2.0 Full-Text:
> >
> >
> > Now I'm part of the task force editing this draft so your comments  
> > are very much welcomed.
> >
> > -- J.D.
> >
> >
> >
> >
> > I've implemented just a few queries (Boolean, Term, FilteredQuery,
> > BoostingQuery ...) but other queries are fairly trivial to add.
> > At this stage I am more interested in feedback on parser design/ 
> > approach
> > rather than trying to achieve complete coverage of all the Lucene  
> > Query
> > types or debating the choice of tag names.
> >
> > Please see the readme.txt in the package for more details.
> >
> > Cheers
> > Mark

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message