spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Davidson (JIRA)" <>
Subject [jira] [Commented] (SPARK-6373) Add SSL/TLS for the Netty based BlockTransferService
Date Sun, 22 Mar 2015 21:51:10 GMT


Aaron Davidson commented on SPARK-6373:

Thanks for taking a look at this. The API and modifications needed for this are indeed similar
to SPARK-6229, and so I'd like ideally to find a common way to support both of them. My high
level proposal on that thread was to add an "EncryptionHandler" interface which is implemented
by both SASL and SSL, which returns a Netty Handler which will be attached to an appropriate
point in the pipeline. Additionally, the blocking part of the setup can be implemented using
the TransportClientBootstrap mechanism, where either SSL or SASL simply adds a new bootstrap
which waits for the Handler to enter the final state of the initialization.

We could additionally always add the ChunkedWriteHandler, I suspect the overhead is pretty

So from your perspective, would such an API be tenable? Looking at your implementation, it
seems like we could avoid the different pipelines for Client vs Server if we make the above
changes, and would take the SSLFactory via the TransportContext constructor, wrapped inside
an EncryptionHandler. Otherwise, your implementation would remain the same, though.

> Add SSL/TLS for the Netty based BlockTransferService 
> -----------------------------------------------------
>                 Key: SPARK-6373
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Block Manager, Shuffle
>    Affects Versions: 1.2.1
>            Reporter: Jeffrey Turpin
> Add the ability to allow for secure communications (SSL/TLS) for the Netty based BlockTransferService
and the ExternalShuffleClient. This ticket will hopefully start the conversation around potential
designs... Below is a reference to a WIP prototype which implements this functionality (prototype)...
I have attempted to disrupt as little code as possible and tried to follow the current code
structure (for the most part) in the areas I modified. I also studied how Hadoop achieves
encrypted shuffle (

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message