kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Rassaerts <bruno.rassae...@novazone.be>
Subject Re: Encryption on disk
Date Fri, 15 Jan 2016 14:39:58 GMT
Thanks for the input Jim.

We managed to reduce the encryption impact to about 25% by disabling the kafka batch compression
and compressing the messages ourselves before encrypting them one-by-one. However we still
believe we could improve by batch compressing + batch encrypting. 

Can you confirm that in your tests batch compression was disabled ?


> On 14 Jan 2016, at 23:47, Jim Hoagland <jim_hoagland@symantec.com> wrote:
> We did a proof of concept on end-to-end encryption using an approach which
> sounds similar to what you describe.  We blogged about it here:
> http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-p
> roof-concept
> You might want to review what is there to see how it differs from what you
> did.  In our tests, the encryption didn't add as much overhead as we
> thought it would.
> -- Jim
> -- 
> Jim Hoagland, Ph.D.
> Sr. Principal Software Engineer
> Big Data Analytics Team
> Cloud Platform Engineering
> On 1/14/16, 2:23 PM, "Bruno Rassaerts" <bruno.rassaerts@novazone.be> wrote:
>> Hello,
>> In our project we have a very strong requirement to protect all data, all
>> the time. Even when the data is “in-rest” on disk, it needs to be
>> protected.
>> We’ve been trying to figure out how to this with Kafka, and hit some
>> obstacles.
>> One thing we’ve tried to do is to encrypt every message we hand over to
>> kafka. This results in the encrypted messages being written to disk on
>> the brokers.
>> However, the performance of performing encryption has serious performance
>> implications, due to the CPU intensive operation which encryption is, and
>> the fact that batch compression offered by Kafka is not nearly as
>> efficient anymore after encrypting the data. Doing this message by
>> message encryption gives us a performance penalty of about 75%, even if
>> we compress the messages before encryption.
>> What we are looking for is a way to plugin our encryption in two possible
>> locations:
>> 1. As a custom compression algorithm, which would batch compress, and
>> batch encrypt. And get the files stored as such.
>> 2. As a encryption plugin specifically designed for storing the kafka
>> broker files.
>> Is there any way that this can be done using Kafka (0.9), or can somebody
>> point us to the place were we could add this in the Kafka codebase.
>> Thanks,
>> Bruno Rassaerts 

View raw message