From nutch-dev-return-3136-apmail-lucene-nutch-dev-archive=lucene.apache.org@lucene.apache.org Sat Dec 17 16:51:58 2005 Return-Path: Delivered-To: apmail-lucene-nutch-dev-archive@www.apache.org Received: (qmail 93138 invoked from network); 17 Dec 2005 16:51:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 17 Dec 2005 16:51:58 -0000 Received: (qmail 94994 invoked by uid 500); 17 Dec 2005 16:51:58 -0000 Delivered-To: apmail-lucene-nutch-dev-archive@lucene.apache.org Received: (qmail 94639 invoked by uid 500); 17 Dec 2005 16:51:55 -0000 Mailing-List: contact nutch-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: nutch-dev@lucene.apache.org Delivered-To: mailing list nutch-dev@lucene.apache.org Received: (qmail 94627 invoked by uid 500); 17 Dec 2005 16:51:55 -0000 Delivered-To: apmail-incubator-nutch-dev@incubator.apache.org Received: (qmail 94624 invoked by uid 99); 17 Dec 2005 16:51:55 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [192.87.106.226] (HELO ajax.apache.org) (192.87.106.226) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Dec 2005 08:51:55 -0800 Received: from ajax.apache.org (ajax.apache.org [127.0.0.1]) by ajax.apache.org (Postfix) with ESMTP id 67658C9 for ; Sat, 17 Dec 2005 17:51:34 +0100 (CET) Message-ID: <707360147.1134838294391.JavaMail.jira@ajax.apache.org> Date: Sat, 17 Dec 2005 17:51:34 +0100 (CET) From: "Bernhard Messer (JIRA)" To: nutch-dev@incubator.apache.org Subject: [jira] Created: (NUTCH-144) corrupt language identifier tri files and bad language recognition for german Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N corrupt language identifier tri files and bad language recognition for germ= an ---------------------------------------------------------------------------= -- Key: NUTCH-144 URL: http://issues.apache.org/jira/browse/NUTCH-144 Project: Nutch Type: Improvement Versions: 0.8-dev =20 Reporter: Bernhard Messer Priority: Minor Hi, i had a look at the generated language guesser tri files. As far as i can s= ay, several of them (de.ngp, da.ngp, es.ngp) seems to be corrupt which leed= s to bad language recognition ratio. For example the german tri file should= contain the german special characters "=C3=A4", "=C3=B6", "=C3=BC" with th= eir frequency. The text "gr=C3=BCne H=C3=BCte" which is typical german, is = recognized as danish. May be the problem comes from wrong character encodin= g during training. Jerome, could you provide the training files so that the language identifie= r can be retrained ? regards Bernhard --=20 This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira