lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Сергей Бирюков <kapac...@yandex.ru>
Subject Re: Has anyone HunspellStemFilterFactory working?
Date Wed, 14 Nov 2012 11:08:55 GMT
Rob, as regards your "problem"
> 'SET charset'
'charset' word must be replaced with a name-of-character-set (i.e. encoding)
For exampe,  you can write 'SET UTF-8'

BUT...

----

Be careful!
At least for russian language morthology HunspellStemFilterFactory has 
bug(s) in its algorythm.

Simple comparison with original hunspell library shown huge difference.


For example on  Linux x86_64 Ubuntu 12.10

1) INSTALL:
# sudo apt-get install hunspell hunspell-ru


2) TEST with string "мама мыла раму мелом"
(it has a meaning: "mom was_washing frame (with) chalk" ):

2.1) OS hunspell library
# echo "мама мыла раму мелом" | hunspell -d ru_RU -D -m
gives results:
...
     LOADED DICTIONARY:
     /usr/share/hunspell/ru_RU.aff
     /usr/share/hunspell/ru_RU.dic

     мама  -> мама
     мыла  -> мыло | мыть     <<< as noun | as verb
     раму  -> рама
     мелом -> мел

2.2) solr's HunspellStemFilterFactory
config fieldType
     <fieldType name="text_hunspell" class="solr.TextField" 
positionIncrementGap="100">
       <analyzer>
         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
         <filter class="solr.LowerCaseFilterFactory" />
         <filter class="solr.HunspellStemFilterFactory" 
dictionary="ru_RU.dic" affix="ru_RU.aff" ignoreCase="true" />
       </analyzer>
     </fieldType>

gives results:
     мама -> мама | мама         : FAILED:  duplicate words
     мыла -> мыть | мыло         : SUSSECC: all OK
     раму -> рама | расти          : FAILED: second word is wrong and excess
     мелом -> мести | метить | месть | мел  :  FAILED: only last
word is 
correct, other ones are excess

----------

That's why I use a JNA (v3.2.7) binding on original (system) 
libhunspell.so for a long time :)

----
Best regards
   Sergey Biryukov
   Moscow, Russian Federation


14.11.2012 04:18, Rob Koeling wrote:
> If so, would you be willing to share the .dic and .aff files with me?
> When I try to load a dictionary file, Solr is complaining that:
>
> java.lang.RuntimeException: java.io.IOException: Unable to load hunspell
> data! [dictionary=en_GB.dic,affix=en_GB.aff]
>      at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:116)
> .......
> Caused by: java.text.ParseException: The first non-comment line in the
> affix file must be a 'SET charset', was: 'FLAG num'
>      at
> org.apache.lucene.analysis.hunspell.HunspellDictionary.getDictionaryEncoding(HunspellDictionary.java:306)
>      at
> org.apache.lucene.analysis.hunspell.HunspellDictionary.<init>(HunspellDictionary.java:130)
>      at
> org.apache.lucene.analysis.hunspell.HunspellStemFilterFactory.inform(HunspellStemFilterFactory.java:103)
>      ... 46 more
>
> When I change the first line to 'SET charset' it is still not happy. I got
> the dictionary files from the OpenOffice website.
>
> I'm using Solr 4.0 (but had the same problem with 3.6)
>
>    - Rob
>


Mime
View raw message