spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Spark Custom Receiver
Date Tue, 18 Nov 2014 02:12:58 GMT
That is not yet in the roadmap. If there is sufficient demand for it,
we can think about it. But exposing internal communication details in
a API stable manner is usually a little tricky.

TD

On Mon, Nov 17, 2014 at 5:50 PM, Jacob Abraham <abe.jacobs@gmail.com> wrote:
> Thanks TD for your reply. Hacking is fine, but I am not a fan of hacking
> into something that I don't fully understand (law of unintended
> consequences). So, this option is out for me... :) I could setup my own
> actor (I suppose you mean akka actors) and here the problem is that I don't
> understand the actor framework very well. However this is some I could gain
> some knowledge in. No better time to start....
>
> Anyway, is there any plans in the roadmnap for spark streaming where the
> existing actor based messaging will be exposed to the public API?
>
> -abe
>
>
> On Thu, Nov 13, 2014 at 5:30 PM, Tathagata Das <tathagata.das1565@gmail.com>
> wrote:
>>
>> Haha, there is a actor-based messaging path that exists between the
>> driver (see ReceiverTracker) and the Receiver (see
>> ReceiverSupervisorImpl). But none of that is exposed to the public
>> API. So either you could hack in Spark Streaming code to expose that,
>> or it might be better to setup you own actor in the custom receiver,
>> and communicate with another actor in the application driver.
>>
>> Does that make sense?
>>
>> On Thu, Nov 13, 2014 at 5:24 PM, Jacob Abraham <abe.jacobs@gmail.com>
>> wrote:
>> > Hi Folks,
>> >
>> > I have written a custom Spark receiver and in my testing I have found
>> > that
>> > its doing its job properly.
>> >
>> > However, I am wondering if someone could shed some light on how the
>> > "driver"
>> > could query the "receiver" for some information. In other words, how can
>> > I
>> > make the driver talk to the receivers to get some information from it ?
>> > This
>> > information could be something like the state of the receiver or some
>> > statistics about the receiver itself...
>> >
>> > Regards,
>> > -Abe
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message