spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (SPARK-6229) Support encryption in network/common module
Date Wed, 11 Mar 2015 23:55:38 GMT


Marcelo Vanzin commented on SPARK-6229:

/cc [~rxin] [~adav]

Hey Reynold, Aaron,

I started looking at this and it seems like there's no easy way to insert SASL encryption
into the current channel pipeline due to the way the code is organized.

On the client side, {{TransportClientFactory.createClient}} seems like the best candidate,
but it would have to be changed so that {{SaslClientBootstrap}} can add a handler to the channel
pipeline to do the encryption.

On the server side, it seems more complicated. I couldn't find a place where the current SASL
code has access to the channel, so it can't really set up a handler at the same layer as the
client code. At first I thought I could just encrypt things in {{SaslRpcHandler}}, but that
looks fishy since that handler is before all the framing and serialization handlers, and we
kinda want to encrypt after those.

So I'd like to suggest a bigger change that I think would make the code easier to use and
at the same time allow this change: internalize all SASL handling in the network library.

Basically, let the network library decide whether to add SASL to the picture by looking at
the TransportConf; code using the library wouldn't need to care about it anymore (aside from
providing a TransportConf and a SecretKeyHolder).

So, instead of code like this (from StandaloneWorkerShuffleService):

  private val transportContext: TransportContext = {
    val handler = if (useSasl) new SaslRpcHandler(blockHandler, securityManager) else blockHandler
    new TransportContext(transportConf, handler)

You'd have just:

  private val transportContext: TransportContext = new TransportContext(transportConf, securityManager,

And similarly in other places. What do you guys think?

BTW here's some code that does this and is totally transparent to the code creating the RPC
server / client:

And the two implementations:

That's just to give an idea of how it could be done internally in the network library, without
the consumers of the library having to care about it.

But please let me know if I missed something obvious here.

> Support encryption in network/common module
> -------------------------------------------
>                 Key: SPARK-6229
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>            Reporter: Marcelo Vanzin
> After SASL support has been added to network/common, supporting encryption should be
rather simple. Encryption is supported for DIGEST-MD5 and GSSAPI. Since the latter requires
a valid kerberos login to work (and so doesn't really work with executors), encryption would
require the use of DIGEST-MD5.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message