lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chuck Williams" <>
Subject RE: encoding of german analyzer source files
Date Fri, 26 Nov 2004 23:53:41 GMT
Actually, in Netbeans 4.0 although you can tell the editor the encoding
for each file individually, that doesn't solve your problem.  At least
for standard projects, you can only provide the required compiler switch
(-encoding UTF-8) at the project level.  I ran into this same issue
building Lucene and had to set the compiler encoding to be uniformly
UTF-8 for the entire Lucene source tree, which fortunately works.

There are a number of ways to work around this.  E.g., one could put a
non-UTF 8 analyzer into a dependent separate project.

These are of course Netbeans issues and not Lucene issues.  Re. Lucene,
if not automated, there should at least be a readme entry or something
that identifies the necessary file encodings (which could just say that
UTF8 is the required encoding to compile Lucene).  When I first tried to
build Lucene, because of the default ISO-8859-1 encoding, I got errors
concerning illegal character objects.  I didn't know what encoding was
required, or even for sure that my problem was an encoding issue, so I
asked a question on this list.  It would be better if this was more


  > -----Original Message-----
  > From: Murray Altheim []
  > Sent: Friday, November 26, 2004 2:29 PM
  > To: Lucene Developers List
  > Subject: Re: encoding of german analyzer source files
  > Andi Vajda wrote:
  > >>I can tell the NetBeans-IDE the encoding of every single source
  > But the
  > >>problem is that I might not know which the correct encoding is. In
  > case of
  > >>Lucene it is quite clear because it is mentioned in the build.xml
  > But
  > >>what is the situation if someone sends you a stemmer class for
  > for
  > >>Swahili and you do not know in which encoding the author wrote the
  > source.
  > >>Then you can try lots of encodings until the java compiler will be
  > satisfied
  > >>with it. And even then you might not be sure that you used the
  > >>encoding.
  > >
  > >>Therefore it would be great if all Java programmers would agree on
  > same
  > >>encoding of source files (let it be UTF-8, ISO-8859-1 or something
  > really
  > >
  > > Actually, the reason for the change to utf-8 was that for Lucene
  > compile on
  > > Windows with gcj (mingw), the encoding better be utf-8 because of
  > typical
  > > absence of iconv facility there. Therefore, it would be safe to
  > the
  > > swahili stemmer source to also be encoded in utf-8.
  > >
  > > Andi..
  > Andi,
  > It may seem pretty safe to assume from practice, but from the Java
  > programmer's point of view, it's still not. It's perfectly possible
  > that the Swahili file be in UTF-8 or UTF-16, little-endian or big-
  > endian, or perhaps some other encoding we don't even know about.
  > A minor point I was trying to make is that absent some external
  > mechanism, there's really *no way* to know the encoding of a file.
  > You can sniff the first few bytes (which is what is recommended
  > in the XML 1.0 spec, you can see how they do it there), but making
  > such an assumption may lead to program failure if the assumption
  > is incorrect.
  >    Extensible Markup Language (XML) 1.0 (Third Edition)
  >    Appendix F Autodetection of Character Encodings
  > The suggestions there are pretty usable for files that have nothing
  > to do with XML.
  > I don't know how many people on this list are familiar with
  > O'Reilly's "CJKV Information Processing" (with the puffer fish on
  > the cover), which opened up my eyes to a new world. After reading
  > it I got a terrible fright and couldn't sleep for weeks.
  >    "CJKV Information Processing: Chinese, Japanese, Korean
  >       & Vietnamese Computing", by Ken Lunde, O'Reilly Publishing.
  > 0676059?v=glance&vi=reviews
  > Murray
  > Murray Altheim
  > Knowledge Media Institute
  > The Open University, Milton Keynes, Bucks, MK7 6AA, UK
  >    [International Committee of the Red Cross director] Kraehenbuhl
  >    pointed out that complying with international humanitarian law
  >    was "an obligation, not an option", for all sides of the
  >    "If these rules or any other applicable rules of international
  >    humanitarian law are violated, the persons responsible must be
  >    held accountable for their actions," he said. -- BBC News
  >   "In my judgment, this new paradigm [the War on Terror] renders
  >    obsolete Geneva's strict limitations on questioning of enemy
  >    prisoners and renders quaint some of its provisions [...]
  >    Your determination [that the Geneva Conventions] does not apply
  >    would create a reasonable basis in law that [the War Crimes Act]
  >    does not apply, which would provide a solid defense to any future
  >    prosecution." -- Alberto Gonzalez, appointed US Attorney General,
  >    and likely Supreme Court nominee, in a memo to George W. Bush
  > To unsubscribe, e-mail:
  > For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message