spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Snively <>
Subject Roadblock with Spark 0.8.0 ActorStream
Date Wed, 02 Oct 2013 17:37:10 GMT
Hi everyone,

I think I have a knack for trying things a bit outside the box. :-)

I'm trying to develop a TLS-aware streaming TCP server that creates a Spark Stream for each
incoming connection. Since Akka 2.2's new IO module has been backported to Akka 2.0.5 thanks
to <>, it seemed the easiest thing to
do would be to use Spark's ActorStream support to create a DStream from the ConnectionHandler
Actor with the Receiver trait mixed in, and the Tcp.Received(data) case simply calling pushBlock(data).
My server class just takes a StreamingContext and a Function2[ActorRef, DStream[ByteString],
Unit] and, when a connection is accepted, constructs the DStream, then passes the connection
and DStream to the callback.

I was able to write this, and it compiles OK, but my test throws an IllegalStateException
because deserializing a task is attempting to deserialize an ActorRef, presumably the TCP
connection, without an ActorSystem in scope. The exception message is helpful:

java.lang.IllegalStateException: Trying to deserialize a serialized ActorRef without an ActorSystem
in scope. Use 'akka.serialization.Serialization.currentSystem.withValue(system) { ... }'

However, my attempts to, e.g. wrap my callback invocation in this are unsuccessful, since
I don't have access to an ExtendedActorSystem to pass in.

A small project with code and my failing test is at <>.
If someone has a bit of time to help me look into this, I'd be grateful. Note that this is
a very generic task that I think more people than just me would be interested in. :-)

View raw message