kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Rassaerts <bruno.rassae...@novazone.be>
Subject Re: Encryption at Rest
Date Tue, 03 May 2016 06:55:52 GMT
From what I understand, when using batch compression in Kafka, the files are stored compressed.
Don’t really see the difference between compression and encryption in that aspect.
If Kafka would support pluggable algorithms for compression (it already supports two), it
would be rather straightforward I guess.


> On 03 May 2016, at 07:02, Christian Csar <christian@csar.us> wrote:
> 
> "We need to be capable of changing encryption keys on regular
> intervals and in case of expected key compromise." is achievable with
> full disk encryption particularly if you are willing to add and remove
> Kafka servers so that you replicate the data to new machines/disks
> with new keys and take the machines with old keys out of use and wipe
> them.
> 
> For the second part of it I would suggest reevaluating your threat
> model since you are looking at a machine that is compromised but not
> compromised enough to be able to read the key from Kafka or to use
> Kafka to read the data.
> 
> While you could add support to encrypt data on the way in and out of
> compression I believe you would need either substantial work in Kafka
> to support rewriting/reencrypting the logfiles (with performance
> penalties) or rotate machines in and out as with full disk encryption.
> Though I'll let someone with more knowledge of the implementation
> comment further on what would be required.
> 
> Christian
> 
> On Mon, May 2, 2016 at 9:41 PM, Bruno Rassaerts
> <bruno.rassaerts@novazone.be> wrote:
>> We did try indeed the last scenario you describe as encrypted disks do not fulfil
our requirements.
>> We need to be capable of changing encryption keys on regular intervals and in case
of expected key compromise.
>> Also, when a running machine is hacked, disk based or file system based encryption
doesn’t offer any protection.
>> 
>> Our goal is indeed to have the content in the broker files encrypted. The problem
is that only way to achieve this is through custom serialisers.
>> This works, but the overhead is quite dramatic as the messages are no longer efficiently
compressed (in batch).
>> Compression in the serialiser, before the encryption, doesn’t really solve the
performance problem.
>> 
>> The best thing for us would be able to encrypt after the batch compression offered
by kafka.
>> The hook to do this is missing in the current implementation.
>> 
>> Bruno
>> 
>>> On 02 May 2016, at 22:46, Tom Brown <tombrown52@gmail.com> wrote:
>>> 
>>> I'm trying to understand your use-case for encrypted data.
>>> 
>>> Does it need to be encrypted only over the wire? This can be accomplished
>>> using TLS encryption (v0.9.0.0+). See
>>> https://issues.apache.org/jira/browse/KAFKA-1690
>>> 
>>> Does it need to be encrypted only when at rest? This can be accomplished
>>> using full disk encryption as others have mentioned.
>>> 
>>> Does it need to be encrypted during both? Use both TLS and full disk
>>> encryption.
>>> 
>>> Does it need to be encrypted fully from end-to-end so even Kafka can't read
>>> it? Since Kafka shouldn't be able to know the contents, the key should not
>>> be known to Kafka. What remains is manually encrypting each message before
>>> giving it to the producer (or by implementing an encrypting serializer).
>>> Either way, each message is still encrypted individually.
>>> 
>>> Have I left out a scenario?
>>> 
>>> --Tom
>>> 
>>> 
>>> On Mon, May 2, 2016 at 2:01 PM, Bruno Rassaerts <bruno.rassaerts@novazone.be
>>>> wrote:
>>> 
>>>> Hello,
>>>> 
>>>> We tried encrypting the data before sending it to kafka, however this
>>>> makes the compression done by kafka almost impossible.
>>>> Also the performance overhead of encrypting the individual messages was
>>>> quite significant.
>>>> 
>>>> Ideally, a pluggable “compression” algorithm would be best. Where message
>>>> can first be compressed, then encrypted in batch.
>>>> However, the current kafka implementation does not allow this.
>>>> 
>>>> Bruno
>>>> 
>>>>> On 26 Apr 2016, at 19:02, Jim Hoagland <jim_hoagland@symantec.com>
>>>> wrote:
>>>>> 
>>>>> Another option is to encrypt the data before you hand it to Kafka and
>>>> have
>>>>> the downstream decrypt it.  This takes care of on-disk on on-wire
>>>>> encryption.  We did a proof of concept of this:
>>>>> 
>>>>> 
>>>> http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-p
>>>>> roof-concept
>>>>> 
>>>>> ( http://symc.ly/1pC2CEG )
>>>>> 
>>>>> -- Jim
>>>>> 
>>>>> On 4/25/16, 11:39 AM, "David Buschman" <david.buschman@timeli.io>
wrote:
>>>>> 
>>>>>> Kafka handles messages which are compose of an array of bytes. Kafka
>>>> does
>>>>>> not care what is in those byte arrays.
>>>>>> 
>>>>>> You could use a custom Serializer and Deserializer to encrypt and
>>>> decrypt
>>>>>> the data from with your application(s) easily enough.
>>>>>> 
>>>>>> This give the benefit of having encryption at rest and over the wire.
>>>> Two
>>>>>> birds, one stone.
>>>>>> 
>>>>>> DaVe.
>>>>>> 
>>>>>> 
>>>>>>> On Apr 25, 2016, at 2:14 AM, Jens Rantil <jens.rantil@tink.se>
wrote:
>>>>>>> 
>>>>>>> IMHO, I think that responsibility should lie on the file system,
not
>>>>>>> Kafka.
>>>>>>> Feels like a waste of time and double work to implement that
unless
>>>>>>> there's
>>>>>>> a really good reason for it. Let's try to keep Kafka a focused
product
>>>>>>> that
>>>>>>> does one thing well.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Jens
>>>>>>> 
>>>>>>> On Fri, Apr 22, 2016 at 3:31 AM Tauzell, Dave
>>>>>>> <Dave.Tauzell@surescripts.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I meant encryption of the data at rest.  We utilize filesytem
>>>>>>>> encryption
>>>>>>>> for other products; just wondering if anything was on the
Kafka
>>>>>>>> roadmap.
>>>>>>>> 
>>>>>>>> Dave
>>>>>>>> 
>>>>>>>>> On Apr 21, 2016, at 18:12, Martin Gainty <mgainty@hotmail.com>
>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Dave-
>>>>>>>>> so you want username/password credentials to be sent
in response to
>>>> an
>>>>>>>> HTTP Get as clear text?
>>>>>>>>> if not this has been asked and answered with Axishttps://
>>>>>>>> axis.apache.org/axis2/java/rampart/
>>>>>>>>> 
>>>>>>>>> Martin
>>>>>>>>> ______________________________________________
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> From: Dave.Tauzell@surescripts.com
>>>>>>>>>> To: users@kafka.apache.org
>>>>>>>>>> Subject: Encryption at Rest
>>>>>>>>>> Date: Thu, 21 Apr 2016 21:31:56 +0000
>>>>>>>>>> 
>>>>>>>>>> Has there been any discussion or work on at rest
encryption for
>>>>>>>>>> Kafka?
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Dave
>>>>>>>>>> 
>>>>>>>>>> This e-mail and any files transmitted with it are
confidential, may
>>>>>>>> contain sensitive information, and are intended solely for
the use of
>>>>>>>> the
>>>>>>>> individual or entity to whom they are addressed. If you have
received
>>>>>>>> this
>>>>>>>> e-mail in error, please notify the sender by reply e-mail
immediately
>>>>>>>> and
>>>>>>>> destroy all copies of the e-mail and any attachments.
>>>>>>>>> 
>>>>>>>> 
>>>>>>> --
>>>>>>> 
>>>>>>> Jens Rantil
>>>>>>> Backend Developer @ Tink
>>>>>>> 
>>>>>>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>>>>>>> For urgent matters you can reach me at +46-708-84 18 32.
>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 


Mime
View raw message