kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <joe.st...@stealth.ly>
Subject Re: [DISCUSS] Kafka Security Specific Features
Date Thu, 05 Jun 2014 16:02:17 GMT
Raja, you need to sign an ICLA http://www.apache.org/licenses/icla.txt once
that is on file your user can get permed to contribute.

I think securing communication to "offset & broker management source" which
can be a zookeeper implementation is important.  I will elaborate more on
that with the other edits I have for the page.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/


On Thu, Jun 5, 2014 at 11:56 AM, Rajasekar Elango <relango@salesforce.com>
wrote:

> Hi Jay,
>
> Thanks for putting together a spec for security.
>
> Joe,
>
> Looks "Securing zookeeper.." part has been deleted from assumptions
> section. communication with zookeeper need to be secured as well to make
> entire kafka cluster secure. It may or may not require changes to kafka.
> But it's good to have it in spec.
>
> I could not find a link to edit the page after login into wiki. Do I need
> any special permission to make edits?
>
> Thanks,
> Raja.
>
>
> On Wed, Jun 4, 2014 at 8:57 PM, Joe Stein <joe.stein@stealth.ly> wrote:
>
> > I like the idea of working on the spec and prioritizing. I will update
> the
> > wiki.
> >
> > - Joestein
> >
> >
> > On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps <jay.kreps@gmail.com> wrote:
> >
> > > Hey Joe,
> > >
> > > Thanks for kicking this discussion off! I totally agree that for
> > something
> > > that acts as a central message broker security is critical feature. I
> > think
> > > a number of people have been interested in this topic and several
> people
> > > have put effort into special purpose security efforts.
> > >
> > > Since most the LinkedIn folks are working on the consumer right now I
> > think
> > > this would be a great project for any other interested people to take
> on.
> > > There are some challenges in doing these things distributed but it can
> > also
> > > be a lot of fun.
> > >
> > > I think a good first step would be to get a written plan we can all
> agree
> > > on for how things should work. Then we can break things down into
> chunks
> > > that can be done independently while still aiming at a good end state.
> > >
> > > I had tried to write up some notes that summarized at least the
> thoughts
> > I
> > > had had on security:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Security
> > >
> > > What do you think of that?
> > >
> > > One assumption I had (which may be incorrect) is that although we want
> > all
> > > the things in your list, the two most pressing would be authentication
> > and
> > > authorization, and that was all that write up covered. You have more
> > > experience in this domain, so I wonder how you would prioritize?
> > >
> > > Those notes are really sketchy, so I think the first goal I would have
> > > would be to get to a real spec we can all agree on and discuss. A lot
> of
> > > the security stuff has a high human interaction element and needs to
> work
> > > in pretty different domains and different companies so getting this
> kind
> > of
> > > review is important.
> > >
> > > -Jay
> > >
> > >
> > > On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein <joe.stein@stealth.ly>
> wrote:
> > >
> > > > Hi,I wanted to re-ignite the discussion around Apache Kafka Security.
> > >  This
> > > > is a huge bottleneck (non-starter in some cases) for a lot of
> > > organizations
> > > > (due to regulatory, compliance and other requirements). Below are my
> > > > suggestions for specific changes in Kafka to accommodate security
> > > > requirements.  This comes from what folks are doing "in the wild" to
> > > > workaround and implement security with Kafka as it is today and also
> > > what I
> > > > have discovered from organizations about their blockers. It also
> picks
> > up
> > > > from the wiki (which I should have time to update later in the week
> > based
> > > > on the below and feedback from the thread).
> > > >
> > > > 1) Transport Layer Security (i.e. SSL)
> > > >
> > > > This also includes client authentication in addition to in-transit
> > > security
> > > > layer.  This work has been picked up here
> > > > https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate
> any
> > > > thoughts, comments, feedback, tomatoes, whatever for this patch.  It
> > is a
> > > > pickup from the fork of the work first done here
> > > > https://github.com/relango/kafka/tree/kafka_security.
> > > >
> > > > 2) Data encryption at rest.
> > > >
> > > > This is very important and something that can be facilitated within
> the
> > > > wire protocol. It requires an additional map data structure for the
> > > > "encrypted [data encryption key]". With this map (either in your
> object
> > > or
> > > > in the wire protocol) you can store the dynamically generated
> symmetric
> > > key
> > > > (for each message) and then encrypt the data using that dynamically
> > > > generated key.  You then encrypt the encryption key using each public
> > key
> > > > for whom is expected to be able to decrypt the encryption key to then
> > > > decrypt the message.  For each public key encrypted symmetric key
> > (which
> > > is
> > > > now the "encrypted [data encryption key]" along with which public key
> > it
> > > > was encrypted with for (so a map of [publicKey] =
> > > > encryptedDataEncryptionKey) as a chain.   Other patterns can be
> > > implemented
> > > > but this is a pretty standard digital enveloping [0] pattern with
> only
> > 1
> > > > field added. Other patterns should be able to use that field to-do
> > their
> > > > implementation too.
> > > >
> > > > 3) Non-repudiation and long term non-repudiation.
> > > >
> > > > Non-repudiation is proving data hasn't changed.  This is often (if
> not
> > > > always) done with x509 public certificates (chained to a certificate
> > > > authority).
> > > >
> > > > Long term non-repudiation is what happens when the certificates of
> the
> > > > certificate authority are expired (or revoked) and everything ever
> > signed
> > > > (ever) with that certificate's public key then becomes "no longer
> > > provable
> > > > as ever being authentic".  That is where RFC3126 [1] and RFC3161 [2]
> > come
> > > > in (or worm drives [hardware], etc).
> > > >
> > > > For either (or both) of these it is an operation of the encryptor to
> > > > sign/hash the data (with or without third party trusted timestap of
> the
> > > > signing event) and encrypt that with their own private key and
> > distribute
> > > > the results (before and after encrypting if required) along with
> their
> > > > public key. This structure is a bit more complex but feasible, it is
> a
> > > map
> > > > of digital signature formats and the chain of dig sig attestations.
> >  The
> > > > map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) and
> > > then
> > > > a list of map where that key is "purpose" of signature (what your
> > > attesting
> > > > too).  As a sibling field to the list another field for "the
> attester"
> > as
> > > > bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures).
> > > >
> > > > 4) Authorization
> > > >
> > > > We should have a policy of "404" for data, topics, partitions (etc)
> if
> > > > authenticated connections do not have access.  In "secure mode" any
> non
> > > > authenticated connections should get a "404" type message on
> > everything.
> > > > Knowing "something is there" is a security risk in many uses cases.
>  So
> > > if
> > > > you don't have access you don't even see it.  Baking "that" into
> Kafka
> > > > along with some interface for entitlement (access management) systems
> > > > (pretty standard) is all that I think needs to be done to the core
> > > project.
> > > >  I want to tackle item later in the year after summer after the other
> > > three
> > > > are complete.
> > > >
> > > > I look forward to thoughts on this and anyone else interested in
> > working
> > > > with us on these items.
> > > >
> > > > [0]
> > > >
> > > >
> > >
> >
> http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm
> > > > [1] http://tools.ietf.org/html/rfc3126
> > > > [2] http://tools.ietf.org/html/rfc3161
> > > > [3]
> > > >
> > > >
> > >
> >
> http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm
> > > > [4] http://en.wikipedia.org/wiki/XML_Signature
> > > > [5] http://en.wikipedia.org/wiki/PKCS_12
> > > >
> > > > /*******************************************
> > > >  Joe Stein
> > > >  Founder, Principal Consultant
> > > >  Big Data Open Source Security LLC
> > > >  http://www.stealth.ly
> > > >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > > ********************************************/
> > > >
> > >
> >
>
>
>
> --
> Thanks,
> Raja.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message