lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Morley" <>
Subject RE: German Compound Splitter words.fst causing problems.
Date Thu, 26 Mar 2015 11:51:04 GMT
Thanks for the tip Markus.  We are using this filter to decompound German words.  Update: I
am on the path to victory.  The words.fst file is actually built by the plugin, however there
is a basic input&output file format mismatch (at the byte level) that doesn't occur with
4.0.  As soon as you try to use lucene core 4.1 with this particular plugin, it breaks with
the same error I was getting.  The FST code in lucene says clearly that there is no guaranteed
backward compatibility, so there you have it.  I'm probably going to need to incorporate some
older code from lucene and/or figure out how to make the plugin work with the new lucene code.


-----Original Message-----
From: Markus Jelsma [] 
Sent: Wednesday, March 25, 2015 6:15 PM
Subject: RE: German Compound Splitter words.fst causing problems.

Hello Chris - i don't know that token filter you mention but i would like to recommend Lucene's
HyphenationCompoundWordTokenFilter. It works reasonably well if you provide the hyphenation
rules and a dictionary. It has some flaws such as decompounding to irrelevant subwords, overlapping
subwords or to subwords that do not form the whole compound word (minus genitives),  but these
can be fixed.

-----Original message-----
> From:Chris Morley <>
> Sent: Wednesday 25th March 2015 17:59
> To:
> Subject: German Compound Splitter words.fst causing problems.
> Hello, Chris Morley here, of I am working on the German compound-splitter
by Dawid Weiss. 
>   I tried to "upgrade" the words.fst file that comes with the German compound-splitter
using Solr 3.5, but it doesn't work. Below is the IndexNotFoundException that I get.
>  cmorley@Caracal01:~/Work/oss/git/apache-solr-3.5.0$ java -cp 
> lucene/build/lucene-core-3.5-SNAPSHOT.jar org.apache.lucene.index.IndexUpgrader wordsFst
 Exception in thread "main" org.apache.lucene.index.IndexNotFoundException:
>                  at org.apache.lucene.index.IndexUpgrader.upgrade(
>                  at 
> org.apache.lucene.index.IndexUpgrader.main(
>  The reason I'm attempting this at all is due to the answer here,,
which says to do the upgrade in a two step process, first using Solr 3.5, and then the latest
Solr version (4.10.3).  When I try this running the unit tests for my modified German compound-splitter
I'm getting this same type of error.  The thing is, this is an FST, not an index, which is
a little confusing.  The reason why I'm following this answer though, is because I'm getting
that exact same message when trying to build the (modified) project with the point
at which it tries to load in words.fst. Below.
>  [main] ERROR - Format
version is not supported (resource:
0 (needs to be between 3 and 4). This version of Lucene only supports indexes created with
release 3.0 and later.  Failed to initialize static data structures for German compound splitter.
>  Thanks,
>  -Chris.

View raw message