lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chuck Williams <>
Subject Re: Attached proposed modifications to Lucene 2.0 to support Field.Store.Encrypted
Date Tue, 05 Dec 2006 22:28:23 GMT

Mike Klaas wrote on 12/05/2006 11:38 AM:
> On 12/5/06, negrinv <> wrote:
>> Chris Hostetter wrote:
>> > If the code was not already in the core, and someone asked about
>> adding it
>> > I would argue against doing so on the grounds that some helpfull
>> utility
>> > methods (possibly in a contrib) would be just as usefull, and would
>> have
>> > no performance cost for people who don't care about compression.
>> >
>> Perhaps, if you look at compression on its own, but once you see
>> compression
>> in the context of all the other field options it makes sense to have it
>> added to Lucene, it's about having everything in one place for ease of
>> implementation that offsets the performance issue, in my opinion.
> Note that built-in compression is deprecated, for similar reasons as
> are being given for the encrypted fields.

Built-in compression is also memory-hungry and slow due to the copying
it does.  External compression is much faster, especially if you extend
Field binary values to support a binary length parameter (which I
submitted a patch for a long time ago).

Here is another argument against adding Field encryption to the lucene
core.  Changes in index format make life complex for any implementations
that deal with index files directly.  There are a number of Lucene
sister projects that do this, plus a number of applications.

I have a fast bulk updater that directly manipulates index files and am
busy upgrading it right now to the 2.1 index format with lockless
commits (which is not fully documented in the new index file formats, by
the way, e.g. the <segment>N.sM separate norm files are missing).  It's
a pain.  In general, I think changes to Lucene index format should only
be driven by compelling benefits.  Moving encryption from external to
internal to get a minor application simplification is not sufficiently
compelling to me.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message