kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Rassaerts <bruno.rassae...@novazone.be>
Subject Re: Encryption at Rest
Date Mon, 02 May 2016 20:01:38 GMT
Hello,

We tried encrypting the data before sending it to kafka, however this makes the compression
done by kafka almost impossible.
Also the performance overhead of encrypting the individual messages was quite significant.

Ideally, a pluggable “compression” algorithm would be best. Where message can first be
compressed, then encrypted in batch.
However, the current kafka implementation does not allow this.

Bruno

> On 26 Apr 2016, at 19:02, Jim Hoagland <jim_hoagland@symantec.com> wrote:
> 
> Another option is to encrypt the data before you hand it to Kafka and have
> the downstream decrypt it.  This takes care of on-disk on on-wire
> encryption.  We did a proof of concept of this:
> 
> http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-p
> roof-concept
> 
>  ( http://symc.ly/1pC2CEG )
> 
> -- Jim
> 
> On 4/25/16, 11:39 AM, "David Buschman" <david.buschman@timeli.io> wrote:
> 
>> Kafka handles messages which are compose of an array of bytes. Kafka does
>> not care what is in those byte arrays.
>> 
>> You could use a custom Serializer and Deserializer to encrypt and decrypt
>> the data from with your application(s) easily enough.
>> 
>> This give the benefit of having encryption at rest and over the wire. Two
>> birds, one stone.
>> 
>> DaVe.
>> 
>> 
>>> On Apr 25, 2016, at 2:14 AM, Jens Rantil <jens.rantil@tink.se> wrote:
>>> 
>>> IMHO, I think that responsibility should lie on the file system, not
>>> Kafka.
>>> Feels like a waste of time and double work to implement that unless
>>> there's
>>> a really good reason for it. Let's try to keep Kafka a focused product
>>> that
>>> does one thing well.
>>> 
>>> Cheers,
>>> Jens
>>> 
>>> On Fri, Apr 22, 2016 at 3:31 AM Tauzell, Dave
>>> <Dave.Tauzell@surescripts.com>
>>> wrote:
>>> 
>>>> I meant encryption of the data at rest.  We utilize filesytem
>>>> encryption
>>>> for other products; just wondering if anything was on the Kafka
>>>> roadmap.
>>>> 
>>>> Dave
>>>> 
>>>>> On Apr 21, 2016, at 18:12, Martin Gainty <mgainty@hotmail.com>
wrote:
>>>>> 
>>>>> Dave-
>>>>> so you want username/password credentials to be sent in response to an
>>>> HTTP Get as clear text?
>>>>> if not this has been asked and answered with Axishttps://
>>>> axis.apache.org/axis2/java/rampart/
>>>>> 
>>>>> Martin
>>>>> ______________________________________________
>>>>> 
>>>>> 
>>>>> 
>>>>>> From: Dave.Tauzell@surescripts.com
>>>>>> To: users@kafka.apache.org
>>>>>> Subject: Encryption at Rest
>>>>>> Date: Thu, 21 Apr 2016 21:31:56 +0000
>>>>>> 
>>>>>> Has there been any discussion or work on at rest encryption for
>>>>>> Kafka?
>>>>>> 
>>>>>> Thanks,
>>>>>> Dave
>>>>>> 
>>>>>> This e-mail and any files transmitted with it are confidential, may
>>>> contain sensitive information, and are intended solely for the use of
>>>> the
>>>> individual or entity to whom they are addressed. If you have received
>>>> this
>>>> e-mail in error, please notify the sender by reply e-mail immediately
>>>> and
>>>> destroy all copies of the e-mail and any attachments.
>>>>> 
>>>> 
>>> -- 
>>> 
>>> Jens Rantil
>>> Backend Developer @ Tink
>>> 
>>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>>> For urgent matters you can reach me at +46-708-84 18 32.
>> 
> 


Mime
View raw message