lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Encryption at lucene index
Date Fri, 11 Aug 2017 17:05:54 GMT
Encrypting the _tokens_ inevitably leads to reduced capabilities BTW.
Trivial example:
I have these tokens in my index
run
runner
running
runs

Any non-trivial encryption algorithm will not encrypt the first three
letters "run" identically in all three so searching for run* simply
won't work.

As you can see, there's quite a bit of back-and-forth with that JIRA
and it is pretty much been abandoned.

Best,
Erick

On Thu, Aug 10, 2017 at 11:17 PM, Kumaran Ramasubramanian
<kums.134@gmail.com> wrote:
> Hi Ishan, thank you :-)
>
> -
> -
> Kumaran R
>
>
>
> On Mon, Aug 7, 2017 at 10:53 PM, Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
>> Harry Ochiai (Hitachi) has some index encryption solution,
>> https://www.slideshare.net/maggon/securing-solr-search-data-in-the-cloud
>> I think it is proprietary, but I'm not sure. Maybe more googling might help
>> find the exact page where his solution is described.
>>
>> On Mon, Aug 7, 2017 at 9:59 PM, Kumaran Ramasubramanian <
>> kums.134@gmail.com>
>> wrote:
>>
>> > Hi Erick, i want to encrypt some fields of an document which has personal
>> > identifiable information ( both indexed and stored data)... for eg:
>> email,
>> > mobilenumber etc.. i am able to find LUCENE-6966 alone while googling
>> it..
>> > any related pointers in solr or latest lucene version?
>> >
>> >
>> > -
>> > -
>> > Kumaran R
>> >
>> > On Mon, Aug 7, 2017 at 9:52 PM, Erick Erickson <erickerickson@gmail.com>
>> > wrote:
>> >
>> > > No, since you haven't defined what you want to encrypt, what your
>> > > requirements are, what you hope to get out of "encryption" etc.
>> > >
>> > > Put the index on an encrypting filesystem and forget about it if you
>> > > possibly can, because anything else is a significant amount of work.
>> > > To encrypt the searchable tokens on a per-user basis in memory is a
>> > > _lot_ of work. It depends on your security needs.
>> > >
>> > > Otherwise, as I said, please ask specific questions as the topic is
>> > > quite large, much too large to conduct a seminar through the user's
>> > > list.
>> > >
>> > > Best,
>> > > Erick
>> > >
>> > > On Mon, Aug 7, 2017 at 9:07 AM, Kumaran Ramasubramanian
>> > > <kums.134@gmail.com> wrote:
>> > > > Hi Erick,
>> > > >
>> > > >     Thanks for the information. Any pointers about encryption options
>> > in
>> > > > solr?
>> > > >
>> > > >
>> > > > --
>> > > > Kumaran R
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Aug 7, 2017 at 9:17 PM, Erick Erickson <
>> > erickerickson@gmail.com>
>> > > > wrote:
>> > > >
>> > > >> Encryption in Solr has a bunch of ramifications. Do you care about
>> > > >>
>> > > >> - encryption at rest or in memory?
>> > > >> - encrypting the _searchable_ tokens?
>> > > >> - encrypting the searchable tokens per-user?
>> > > >> - encrypting the stored data (which a filter won't do BTW).
>> > > >>
>> > > >> It's actually a fairly complex topic the discussion at LUCENE-6966
>> > > >> outlines much of it. Please ask specific questions as you research
>> the
>> > > >> topic. One  per-user encryption package that I know of is by Hitachi
>> > > >> Solutions (commercial) and it explicitly does _not_ support, for
>> > > >> instance, wildcards (there are other limitations too). See:
>> > > >> http://www.hitachi-solutions.com/securesearch/
>> > > >>
>> > > >> Most of the time when people ask for encryption they soon discover
>> > > >> it's much more difficult than they imagine and settle for just
>> putting
>> > > >> the indexes on an encrypting file system. When they move beyond
that
>> > > >> it gets complex and you'd be well advised to consult with Solr
>> > > >> security experts.
>> > > >>
>> > > >> Best,
>> > > >> Erick
>> > > >>
>> > > >> On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian
>> > > >> <kums.134@gmail.com> wrote:
>> > > >> > Hi All,
>> > > >> >
>> > > >> >
>> > > >> > After looking at all below discussions, i have one doubt
which may
>> > be
>> > > >> silly
>> > > >> > or novice but i want to throw this to lucene user list.
>> > > >> >
>> > > >> > if we have encryption layer included in our analyzer's flow
of
>> > filters
>> > > >> like
>> > > >> > EncryptionFilter to control field-level encryption. what
are the
>> > > >> > consequences ? am i missing anything basic?
>> > > >> >
>> > > >> > Thanks in advance..
>> > > >> >
>> > > >> >
>> > > >> > Related links:
>> > > >> >
>> > > >> > https://issues.apache.org/jira/browse/LUCENE-2228 : AES Encrypted
>> > > >> Directory
>> > > >> > - in lucene 3.x
>> > > >> >
>> > > >> > https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec
for
>> > > >> index-level
>> > > >> > encryption - at codec level, to have control on which column
/
>> field
>> > > have
>> > > >> >  personal identifiable information
>> > > >> >
>> > > >> > https://security.stackexchange.com/questions/
>> > > 111153/is-a-lucene-search-
>> > > >> index-effectively-a-backdoor-for-field-level-encryption
>> > > >> >
>> > > >> >
>> > > >> > A decent encrypting algorithm will not produce, say, the
same
>> first
>> > > >> portion
>> > > >> >> for two tokens that start with the same letters. So wildcard
>> > searches
>> > > >> won't
>> > > >> >> work. Consider "runs", "running", "runner". A search
on "run*"
>> > would
>> > > be
>> > > >> >> expected to match all three, but wouldn't unless the
encryption
>> > were
>> > > so
>> > > >> >> trivial as to be useless. Similar issues arise with sorting.
>> "More
>> > > Like
>> > > >> >> This" would be unreliable. There are many other features
of a
>> > robust
>> > > >> search
>> > > >> >> engine that would be impacted, and an index with encrypted
terms
>> > > would
>> > > >> be
>> > > >> >> useful for only exact matches, which usually results
in a poor
>> > search
>> > > >> >> experience.
>> > > >> >
>> > > >> >
>> > > >> > https://stackoverflow.com/questions/36604551/adding-
>> > > >> encryption-to-solr-lucene-indexes
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > --
>> > > >> > Kumaran R
>> > > >>
>> > > >> ------------------------------------------------------------
>> ---------
>> > > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> > > >>
>> > > >>
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> > >
>> > >
>> >
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message