kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sönke Liebau <soenke.lie...@opencore.com.INVALID>
Subject Discussion on requirements for Data Encryption functionality in Kafka (KIP-317)
Date Tue, 02 Oct 2018 11:19:36 GMT
Hi all,

I have created KIP-317 [1] a while ago, which outlines an implementation
proposal to add transparent data encryption functionality to Kafka. The KIP
in its current form is somewhat rigid in its implementation, I will rework
this to become extensible over the next few days to allow for additional

I have discussed the current method of providing keys with a colleague and
while we agree that this is a valid use case for some people, there are
certainly a lot of other valid use cases out there as well.
To ensure that the initial implementation provides the necessary
flexibility I'd like some feedback from the community on what requirements
they would have around data encryption and key management.

The following questions should serve as a starting point for the
discussion, please feel free to address anything that comes to mind which I
have not mentioned here:

- Should encryption be configurable rather on the client or on the broker
and be pushed down to the client?
- Where should keys be stored?
- How much flexibility around keys is necessary - is there for example a
use case that would decide on a per message basis which key to use?
(imagine a topic containing top secret, secret and public data with three
different keys)
- Do we need functionality to prohibit publishing unencrypted messages to
topics based on that topics setup?

Of course the mailing list is the first place that discussions like these
should take place, but sometimes I find a face to face discussion can be
quite useful as well, especially when discussing non-trivial topics (like
encryption). I have reached out to the organizers of the upcoming Kafka
Summit in SF and there might be a chance for us to get a room with a
whiteboard at some point (probably during lunch, when the room is otherwise
unused). Would people be interested in meeting up for 20 minutes to discuss
this in person? I'd be happy to provide a summary on the mailing list
afterwards of course.

Look forward to hearing from all of you!

Best regards,


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message