kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Rassaerts <bruno.rassae...@novazone.be>
Subject Encryption on disk
Date Thu, 14 Jan 2016 22:23:11 GMT

In our project we have a very strong requirement to protect all data, all the time. Even when
the data is “in-rest” on disk, it needs to be protected.
We’ve been trying to figure out how to this with Kafka, and hit some obstacles.

One thing we’ve tried to do is to encrypt every message we hand over to kafka. This results
in the encrypted messages being written to disk on the brokers.
However, the performance of performing encryption has serious performance implications, due
to the CPU intensive operation which encryption is, and the fact that batch compression offered
by Kafka is not nearly as efficient anymore after encrypting the data. Doing this message
by message encryption gives us a performance penalty of about 75%, even if we compress the
messages before encryption.

What we are looking for is a way to plugin our encryption in two possible locations:

1. As a custom compression algorithm, which would batch compress, and batch encrypt. And get
the files stored as such.
2. As a encryption plugin specifically designed for storing the kafka broker files.

Is there any way that this can be done using Kafka (0.9), or can somebody point us to the
place were we could add this in the Kafka codebase.

Bruno Rassaerts 
View raw message