kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Rassaerts <bruno.rassae...@novazone.be>
Subject Re: Kafka encryption
Date Mon, 06 Jun 2016 08:51:13 GMT
Indeed to get proper performance, messages need to be batched before encryption.
However, this is not that straightforward to implement and Kafka has already a very good batching
algorithm.
For example, when do you decide to no longer wait for additional messages and send a non-full
batch ? Not that obvious.

Ideally, we would like kafka clients to encrypt/decrypt the compressed batches of kafka.
That seems like the ideal place to do this.


> On 03 Jun 2016, at 07:27, Jim Hoagland <jim_hoagland@symantec.com> wrote:
> 
> I'm hesitant to cite it because it wasn't really a proper benchmark, but
> with the end-to-end encryption through Kafka proof of concept described at
> http://symc.ly/1pC2CEG, doing the encryption added only 26% to the time
> taken to send messages and only 6% to the time taken to consume messages.
> This is with batching 200 300-byte messages together for encryption.  More
> details are in the blog post.
> 
> Personally I think that encrypting sensitive data before handing it to
> Kafka (or at least before it leaves the producing box) just makes sense to
> do if the situation allows.  The Kafka installation wouldn't be able
> reveal the data even if its systems and networks are compromised because
> it never sees the data in the clear and doesn't know how to decrypt it.
> In the way we set it up, someone would need the recipient's RSA private
> key to decrypt (or would need to have compromised a decrypting system).
> 
> -- Jim
> 
> 
> On 6/2/16, 2:56 AM, "Tom Crayford" <tcrayford@heroku.com> wrote:
> 
>> Filesystem encryption is transparent to Kafka. You don't need to use SSL,
>> but your encryption requirements may cause you to need SSL as well.
>> 
>> With regards to compression, without adding at rest encryption to Kafka
>> (which is a very major piece of work, one that for sure requires a KIP and
>> has many, many implications), there's not much to do there. I think it's
>> worth examining your threat models that require encryption on disk without
>> full disk encryption being suitable. Generally compromised broker machines
>> means an attacker will be able to sniff in flight traffic anyway, if the
>> goal is to never leak messages even if an attacker has full control of the
>> broker machine, I'd suggest that that seems pretty impossible under
>> current
>> operating environments.
>> 
>> If the issue is compliance, I'd recommend querying whichever compliance
>> standard you're operating under about the suitability of full disk
>> encryption, and careful thought about encrypting the most sensitive parts
>> of messages. Whilst encryption in the producer and consumer does lead to
>> performance issues and decrease the capability of compression to shrink a
>> dataset, doing partial encryption of messages is easy enough.
>> 
>> Generally we've found that the kinds of uses of Kafka that require in
>> message encryption (alongside full disk encryption and SSL which we
>> provide
>> as standard) don't have such high throughput needs that they worry about
>> compression etc. That clearly isn't true for all use cases though.
>> 
>> Thanks
>> 
>> Tom Crayford
>> Heroku Kafka
>> 
>> On Thursday, 2 June 2016, Gerard Klijs <gerard.klijs@dizzit.com> wrote:
>> 
>>> You could add a header to every message, with information whether it's
>>> encrypted or not, then you don't have to encrypt all the messages, or
>>> you
>>> only do it for some topics.
>>> 
>>> On Thu, Jun 2, 2016 at 6:36 AM Bruno Rassaerts <
>>> bruno.rassaerts@novazone.be <javascript:;>>
>>> wrote:
>>> 
>>>> It works indeed but encrypting individual messages really influences
>>> the
>>>> batch compression done by Kafka.
>>>> Performance drops to about 1/3 of what it is without (even if we
>>> prepare
>>>> the encrypted samples upfront).
>>>> In the end what we going for is only encrypting what we really really
>>> need
>>>> to encrypt, not every message systematically.
>>>> 
>>>>> On 31 May 2016, at 13:00, Gerard Klijs <gerard.klijs@dizzit.com
>>> <javascript:;>> wrote:
>>>>> 
>>>>> If you want system administrators not being able to see the data,
>>> the
>>>> only
>>>>> option is encryption, with only the clients sharing the key (or
>>> whatever
>>>> is
>>>>> used to (de)crypt the data). Like the example from eugene. I don't
>>> know
>>>> the
>>>>> kind of messages you have, but you could always wrap something
>>> around
>>> any
>>>>> (de)serializer your currently using.
>>>>> 
>>>>> On Tue, May 31, 2016 at 12:21 PM Bruno Rassaerts <
>>>>> bruno.rassaerts@novazone.be <javascript:;>> wrote:
>>>>> 
>>>>>> I’ve asked the same question in the past, and disk encryption was
>>>>>> suggested as a solution as well.
>>>>>> However, as far as I know, disk encryption will not prevent your
>>> data
>>> to
>>>>>> be stolen when the machine is compromised.
>>>>>> What we are looking for is even an additional barrier, so that even
>>>> system
>>>>>> administrators do not have access to the data.
>>>>>> Any suggestions ?
>>>>>> 
>>>>>>> On 24 May 2016, at 14:40, Tom Crayford <tcrayford@heroku.com
>>> <javascript:;>> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> There's no encryption at rest. It's recommended to use filesystem
>>>>>>> encryption, or encryption of each individual message before
>>> producing
>>>> it
>>>>>>> for this.
>>>>>>> 
>>>>>>> Only the new producer and consumers have SSL support.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> Tom Crayford
>>>>>>> Heroku Kafka
>>>>>>> 
>>>>>>> On Tue, May 24, 2016 at 11:33 AM, Snehalata Nagaje <
>>>>>>> snehalata.nagaje@harbingergroup.com <javascript:;>>
wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thanks for quick reply.
>>>>>>>> 
>>>>>>>> Do you mean If I see messages in kafka, those will not be
>>> readable?
>>>>>>>> 
>>>>>>>> And also, we are using new producer but old consumer , does
old
>>>> consumer
>>>>>>>> have ssl support?
>>>>>>>> 
>>>>>>>> As mentioned in document, its not there.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Snehalata
>>>>>>>> 
>>>>>>>> ----- Original Message -----
>>>>>>>> From: "Mudit Kumar" <mudit.kumar@askme.in <javascript:;>>
>>>>>>>> To: users@kafka.apache.org <javascript:;>
>>>>>>>> Sent: Tuesday, May 24, 2016 3:53:26 PM
>>>>>>>> Subject: Re: Kafka encryption
>>>>>>>> 
>>>>>>>> Yes,it does that.What specifically you are looking for?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 5/24/16, 3:52 PM, "Snehalata Nagaje" <
>>>>>>>> snehalata.nagaje@harbingergroup.com <javascript:;>>
wrote:
>>>>>>>> 
>>>>>>>>> Hi All,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> We have requirement of encryption in kafka.
>>>>>>>>> 
>>>>>>>>> As per docs, we can configure kafka with ssl, for secured
>>>>>> communication.
>>>>>>>>> 
>>>>>>>>> But does kafka also stores data in encrypted format?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Snehalata
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
> 


Mime
View raw message