lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hacking Bear <>
Subject Multi-lang analyzer? Re: Multiple Language Indexing and Searching
Date Tue, 06 Sep 2005 03:15:15 GMT
 I have the similar problem to deal with. In fact, a lot of times, the 
documents do not have any lanugage information or it may contain text in 
multiple languages. Further, the user would not like to always supply this 
information. Also the user may very well be interested in documents in 
multiple language.
 I think Google and other search engines allow indexing multi-lanugage 
documents. For example, if you google "Java", there will many matched 
documents in lanugages other than English.
 The only assumption we can make is that the document text are converted to 
Unicode before feeding to Lucene.

So I think the solution should be (1) create one index for all lanugage (2) 
add an advisory attribute like "lang" to specify the language of the 
document; if the language is unknown, just leave it empty or set to "ANY"; 
(3) based on the code pages of the upcoming Unicode character, we 
automatically switch among different analyzers to index the fragments of the 
text; (4) during search, unless the user explicitly requesting documents in 
certain language, we return all matches regardless of lanugage.
 I have browsed through the Lucene and contributed source codes, but I 
cannot tell which analyzer is suitable for use (in (3).) While the logic for 
such an analyzer is probably not too complicate, it seems to demand quite 
some Unicode knowledge to create one.
 Is my approach the right one? Is there an analyzer suitable to use?
- HB

 On 9/5/05, Olivier Jaquemet <> wrote: 
> Hi,
> I'd like to go in details regarding issues that occurs when you want to
> index and search contents in multiple languages.
> I have read Lucene in Action book, and many thread on this mailing list,
> the most interesting so far being this one:
> The solution choosen/recommended by Doug Cutting in this message:
> is the number '2/':
> Having one index for all languages one Document per content's language
> with a field specify its language, and using a query filter when 
> searching.
> While I think it is a good solution:
> - If you have N languages, if you search for something in 1 language,
> you are going to search an index N times too large.
> Wouldn't it be better to have N indices for N languages? That way, each
> index could benefit of its specialized analyser, and if you need to
> search in multiple languages, you just need to merge result of those
> differents analyzer.
> - If you have contents in multiple language like we do, and by that I
> don't mean multiple contents each one having its own language, but
> multiple content, each one being in many languages. You are going to
> have a N to 1, Document/content relation in the index.
> As far as update, delete, and search in multiple language are concerned,
> wouldn't it be simpler to alway keep a 1 to 1 Document/content relation
> in an index?
> As you may have guess, my original thought, even before I read those
> thread, was that the solution number 3. might be more flexible/modular
> than the others, of course it also has its drawbacks:
> - performance issue when doing multiple language search, specially when
> merging results of different index.
> - more complex to code
> - other?
> Can you clarify on this?
> What solutions all of you have choosen til now regarding indexing and
> searching of multiple content in multiple language ?
> Thanks!
> Olivier
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message