spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: custom receiver in java
Date Tue, 01 Apr 2014 20:37:18 GMT
Unfortunately, there isnt a good Java-friendly way to define custom
receivers. However, I am currently refactoring the receiver interface to
make it more Java friendly and I hope to get that in Spark 1.0 release.

In the meantime, I would encourage you define the custom receiver in Scala.
If you are not comfortable with Scala, we can write all the core
functionality in Java as functions, and create a skeleton Scala custom
receiver that calls into the Java functions (shown below). Unfortunately,
you have to use scala compiler to compile this.

TD

-----------------------
      class CustomReceiver(parameters: CustomReceiverParameters)
         extends NetworkReceiver[*<type of records>*]
       {
         protected lazy val blocksGenerator: BlockGenerator =
           new BlockGenerator(StorageLevel.MEMORY_ONLY_SER_2)
*         val javaReceiver = new MyJavaReceiver(parameters, blockGenerator)
 // create an instance of java receiver object*

         protected def onStart() = {
           blocksGenerator.start()
*           javaReceiver.start()   // start the java receiver*
         }

         protected def onStop() {
           blocksGenerator.stop()
*           javaReceiver.stop()*
         }
       }




On Tue, Apr 1, 2014 at 7:54 AM, eric perler <ericperler@hotmail.com> wrote:

> i would like to write a custom receiver to receive data from a Tibco RV
> subject
>
> i found this scala example..
>
>
> http://spark.incubator.apache.org/docs/0.8.0/streaming-custom-receivers.html
>
> but i cant seem to find a java example
>
> does anybody know of a good java example for creating a custom receiver
>
> thx
>

Mime
View raw message