lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rao, Vaijanath" <>
Subject RE: Keywords indexing, "top words", and co-occurrence
Date Wed, 23 Dec 2009 06:35:58 GMT


You can write your own hitcollector and in the collector you can do the
required thing. 

Here is the pseudo code for doing this.

Create a class KeywordCountCollector extends TopFieldDocCollector 

In the class override the collect function where you would do something
like this
collect(int doc, int score ) {
	Document document = searcher.doc(doc);
	List<String>keywords=	getKeywordsFromDocument(document)
	for (int i=0;i<keywords.size();i++ ) {
		if (keywordCountMap.contains(keywords.get(i)) {
+ 1);
		}else {

Public Map<String,Integer> getKeywordCountMap() {
	return keywordCountMap;

Hope this helps.

--Thanks and Regards
Vaijanath N. Rao

-----Original Message-----
From: Ted Dunning [] 
Sent: Saturday, December 19, 2009 12:22 AM
Subject: Re: Keywords indexing, "top words", and co-occurrence

Yes, Lucene will help you do this.  It won't do exactly what you want
without some effort on your part.

Sounds like what you want to do is

a) get a book on Lucene and SOLR

b) use standard indexers and a synonym lookup to produce multiple fields
based on the original text and the synonymed text

c) use SOLR's support for faceting to get the counts you are after.

On Fri, Dec 18, 2009 at 6:10 AM, Kaspar Fischer

> Hi everybody,
> I need to do some text analysis and am looking for a software library 
> (in Java, preferably) to use for this. Lucene came to my mind first, 
> but I actually hope that there is some library (based on Lucene, for 
> example) that solves the problems directly.
> What I want to do is the following:
> 1. In documents that get added to the system I need to find keywords 
> from a predefined, fixed set of keywords. For example, the user will 
> make a query for all documents containing the word "traffic" (this 
> word need not be a
> keyword) and I want to show the number of keyword hits in all 
> documents that contain "traffic":
> - car, cars, automobile, automobiles (3)
> - - Mercedes (2)
> - - Ferrari (2)
> - train, trains (4) // one doc contains "TGV", 3 contain "train" or 
> "trains"
> - - TGV (1)
> - - ICE (0)
> - plane, planes (5)
> - - Boeing (4)
> - - Airbus (1)
> In short: I want to count keyword hits in the documents returned by 
> some query. Notice that the keywords are hierarchically organized and 
> may have synonyms ("car" = "cars" = "automobile").
> 2. If the user queries for free-input word A ("hamburger", say) I want

> to find all keywords (from the above hierarchy) that are close to
> in some sense (word-distance or some similar measure of distance in 
> text) and order them by number of occurrence.
> Can this be done in Lucene? Or do you know of any frameworks that 
> achieve such results?
> Regarding to size, I expect the querys (for "traffic" in 1., or
> in 2.) to return at most 500 documents and each document to contain at

> most 50 keywords.
> Many thanks,
> Kaspar

Ted Dunning, CTO

View raw message