lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claude Quézel <cque...@gmail.com>
Subject Why is the {French|Spanish}LightStemmer the default since 3.6?
Date Mon, 28 Jul 2014 15:23:16 GMT
I just migrated my code from version 3.3 to the latest version 4.9.



While running my unit tests, I noticed that the feminine and masculine form
of the french words "ami" and "amie" in french did not stem to the same
token anymore.



I noticed that the french and spanish stemming algorithm had changed from
{French|Spanish}Stemmer to {French|Spanish}LightStemFilter. Code indicates
3.6



To compare both stemming algorithm (FrenchStemmer and FrenchLightStemmer),
I took a 338990 word dictionary and compared the number of resulting tokens.



FrenchStemmer : 54946 (16% of original size)

FrenchLightStemmer: 208649 (62% of original size)



I don't know the common usage of stemmers nor their performances (cpu), but
for my application, it is clear that the FrenchStemmer gives much better
stemming performances than the FrenchLightStemmer.



My question is much more out of curiosity, why is the LightStemmer the
default since 3.6?



Here are the results for the 48 words of the "abaisse" family. The stemmed
value is on the left and the indented values are words that map to the
stemmed value.



FrenchStemmer

--------------------

abaiss

            abaissa

            abaissable

            abaissables

            abaissai

            abaissaient

            abaissais

            abaissait

            abaissant

            abaissante

            abaissantes

            abaissants

            abaissas

            abaissasse

            abaissassent

            abaissasses

            abaissassiez

            abaissassions

            abaisse

            abaisser

            abaissera

            abaisserai

            abaisseraient

            abaisserais

            abaisserait

            abaisseras

            abaisserez

            abaisseriez

            abaisserions

            abaisserons

            abaisseront

            abaisses

            abaissez

            abaissiez

            abaissions

            abaissâmes

            abaissât

            abaissâtes

            abaissèrent

            abaissé

            abaissée

            abaissées

            abaissés

abaissement

            abaissement

            abaissements

abaissent

            abaissent

abaisseur

            abaisseur

            abaisseurs

abaisson

            abaissons





LightFrenchStemmer

--------------------------

abai

            abaissant

            abaissants

            abaissement

            abaissements

abais

            abaisse

            abaisser

            abaisses

            abaissé

            abaissée

            abaissées

            abaissés

abaisa

            abaissa

            abaissas

abaisabl

            abaissable

            abaissables

abaisai

            abaissai

            abaissais

abaisaient

            abaissaient

abaisait

            abaissait

abaisam

            abaissâmes

abaisant

            abaissante

            abaissantes

abaisas

            abaissasse

            abaissasses

abaisasent

            abaissassent

abaisasiez

            abaissassiez

abaisasion

            abaissassions

abaisat

            abaissât

            abaissâtes

abaisent

            abaissent

abaisera

            abaissera

            abaisseras

abaiserai

            abaisserai

            abaisserais

abaiseraient

            abaisseraient

abaiserait

            abaisserait

abaiserent

            abaissèrent

abaiserez

            abaisserez

abaiseriez

            abaisseriez

abaiserion

            abaisserions

abaiseron

            abaisserons

abaiseront

            abaisseront

abaiseu

            abaisseur

            abaisseurs

abaisez

            abaissez

abaisiez

            abaissiez

abaision

            abaissions

abaison

            abaissons

Mime
View raw message