kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Csar <christ...@csar.us>
Subject Re: Encryption at Rest
Date Tue, 03 May 2016 05:02:05 GMT
"We need to be capable of changing encryption keys on regular
intervals and in case of expected key compromise." is achievable with
full disk encryption particularly if you are willing to add and remove
Kafka servers so that you replicate the data to new machines/disks
with new keys and take the machines with old keys out of use and wipe
them.

For the second part of it I would suggest reevaluating your threat
model since you are looking at a machine that is compromised but not
compromised enough to be able to read the key from Kafka or to use
Kafka to read the data.

While you could add support to encrypt data on the way in and out of
compression I believe you would need either substantial work in Kafka
to support rewriting/reencrypting the logfiles (with performance
penalties) or rotate machines in and out as with full disk encryption.
Though I'll let someone with more knowledge of the implementation
comment further on what would be required.

Christian

On Mon, May 2, 2016 at 9:41 PM, Bruno Rassaerts
<bruno.rassaerts@novazone.be> wrote:
> We did try indeed the last scenario you describe as encrypted disks do not fulfil our
requirements.
> We need to be capable of changing encryption keys on regular intervals and in case of
expected key compromise.
> Also, when a running machine is hacked, disk based or file system based encryption doesn’t
offer any protection.
>
> Our goal is indeed to have the content in the broker files encrypted. The problem is
that only way to achieve this is through custom serialisers.
> This works, but the overhead is quite dramatic as the messages are no longer efficiently
compressed (in batch).
> Compression in the serialiser, before the encryption, doesn’t really solve the performance
problem.
>
> The best thing for us would be able to encrypt after the batch compression offered by
kafka.
> The hook to do this is missing in the current implementation.
>
> Bruno
>
>> On 02 May 2016, at 22:46, Tom Brown <tombrown52@gmail.com> wrote:
>>
>> I'm trying to understand your use-case for encrypted data.
>>
>> Does it need to be encrypted only over the wire? This can be accomplished
>> using TLS encryption (v0.9.0.0+). See
>> https://issues.apache.org/jira/browse/KAFKA-1690
>>
>> Does it need to be encrypted only when at rest? This can be accomplished
>> using full disk encryption as others have mentioned.
>>
>> Does it need to be encrypted during both? Use both TLS and full disk
>> encryption.
>>
>> Does it need to be encrypted fully from end-to-end so even Kafka can't read
>> it? Since Kafka shouldn't be able to know the contents, the key should not
>> be known to Kafka. What remains is manually encrypting each message before
>> giving it to the producer (or by implementing an encrypting serializer).
>> Either way, each message is still encrypted individually.
>>
>> Have I left out a scenario?
>>
>> --Tom
>>
>>
>> On Mon, May 2, 2016 at 2:01 PM, Bruno Rassaerts <bruno.rassaerts@novazone.be
>>> wrote:
>>
>>> Hello,
>>>
>>> We tried encrypting the data before sending it to kafka, however this
>>> makes the compression done by kafka almost impossible.
>>> Also the performance overhead of encrypting the individual messages was
>>> quite significant.
>>>
>>> Ideally, a pluggable “compression” algorithm would be best. Where message
>>> can first be compressed, then encrypted in batch.
>>> However, the current kafka implementation does not allow this.
>>>
>>> Bruno
>>>
>>>> On 26 Apr 2016, at 19:02, Jim Hoagland <jim_hoagland@symantec.com>
>>> wrote:
>>>>
>>>> Another option is to encrypt the data before you hand it to Kafka and
>>> have
>>>> the downstream decrypt it.  This takes care of on-disk on on-wire
>>>> encryption.  We did a proof of concept of this:
>>>>
>>>>
>>> http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-p
>>>> roof-concept
>>>>
>>>> ( http://symc.ly/1pC2CEG )
>>>>
>>>> -- Jim
>>>>
>>>> On 4/25/16, 11:39 AM, "David Buschman" <david.buschman@timeli.io> wrote:
>>>>
>>>>> Kafka handles messages which are compose of an array of bytes. Kafka
>>> does
>>>>> not care what is in those byte arrays.
>>>>>
>>>>> You could use a custom Serializer and Deserializer to encrypt and
>>> decrypt
>>>>> the data from with your application(s) easily enough.
>>>>>
>>>>> This give the benefit of having encryption at rest and over the wire.
>>> Two
>>>>> birds, one stone.
>>>>>
>>>>> DaVe.
>>>>>
>>>>>
>>>>>> On Apr 25, 2016, at 2:14 AM, Jens Rantil <jens.rantil@tink.se>
wrote:
>>>>>>
>>>>>> IMHO, I think that responsibility should lie on the file system,
not
>>>>>> Kafka.
>>>>>> Feels like a waste of time and double work to implement that unless
>>>>>> there's
>>>>>> a really good reason for it. Let's try to keep Kafka a focused product
>>>>>> that
>>>>>> does one thing well.
>>>>>>
>>>>>> Cheers,
>>>>>> Jens
>>>>>>
>>>>>> On Fri, Apr 22, 2016 at 3:31 AM Tauzell, Dave
>>>>>> <Dave.Tauzell@surescripts.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I meant encryption of the data at rest.  We utilize filesytem
>>>>>>> encryption
>>>>>>> for other products; just wondering if anything was on the Kafka
>>>>>>> roadmap.
>>>>>>>
>>>>>>> Dave
>>>>>>>
>>>>>>>> On Apr 21, 2016, at 18:12, Martin Gainty <mgainty@hotmail.com>
>>> wrote:
>>>>>>>>
>>>>>>>> Dave-
>>>>>>>> so you want username/password credentials to be sent in response
to
>>> an
>>>>>>> HTTP Get as clear text?
>>>>>>>> if not this has been asked and answered with Axishttps://
>>>>>>> axis.apache.org/axis2/java/rampart/
>>>>>>>>
>>>>>>>> Martin
>>>>>>>> ______________________________________________
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> From: Dave.Tauzell@surescripts.com
>>>>>>>>> To: users@kafka.apache.org
>>>>>>>>> Subject: Encryption at Rest
>>>>>>>>> Date: Thu, 21 Apr 2016 21:31:56 +0000
>>>>>>>>>
>>>>>>>>> Has there been any discussion or work on at rest encryption
for
>>>>>>>>> Kafka?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Dave
>>>>>>>>>
>>>>>>>>> This e-mail and any files transmitted with it are confidential,
may
>>>>>>> contain sensitive information, and are intended solely for the
use of
>>>>>>> the
>>>>>>> individual or entity to whom they are addressed. If you have
received
>>>>>>> this
>>>>>>> e-mail in error, please notify the sender by reply e-mail immediately
>>>>>>> and
>>>>>>> destroy all copies of the e-mail and any attachments.
>>>>>>>>
>>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jens Rantil
>>>>>> Backend Developer @ Tink
>>>>>>
>>>>>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>>>>>> For urgent matters you can reach me at +46-708-84 18 32.
>>>>>
>>>>
>>>
>>>
>

Mime
View raw message