spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andy petrella <andy.petre...@gmail.com>
Subject Re: Spark Streaming Server
Date Wed, 18 Sep 2013 19:55:21 GMT
Hello there,
I hope I get the whole thing correctly... but I think I did something quite
similar that polls the yahoo api for data, which I push into a DStream:
https://github.com/andypetrella/spark-bd/blob/master/src/main/scala/yahoo.scala#L41

Any comments or even concerns (if they are constructive ^^) are more than
welcome.

And, btw, I was thinking the same, this kind of use cases should be enabled
easier in the API.

Cheers
andy



On Wed, Sep 18, 2013 at 9:42 PM, Vadim Chekan <kot.begemot@gmail.com> wrote:

> Hi Paul,
>
> I had the same thoughts when started looking at spark.
> This link might be helpful:
> https://github.com/mesos/spark/blob/master/docs/streaming-custom-receivers.md
>
> Vadim.
>
>
>
> On Tue, Sep 17, 2013 at 6:50 PM, Paul Snively <psnively@icloud.com> wrote:
>
>> Hi everyone!
>>
>> In looking at the Spark Streaming API more closely, it occurs to me that
>> it seems biased toward seeing Spark Streaming processes as consumers of
>> existing services. I'm thinking particularly of socketStream and zeroMQ
>> stream, which want a host/port and Publisher, respectively. But I think
>> this overlooks a whole class of use cases, where I'd like to establish a
>> server that someone can stream data to, and use Spark Streaming to process
>> it. Has any work been done on this front? I suppose, worst case, I could
>> write my own NetworkReceiver that listens on a socket or somesuch, but it'd
>> be nice if I didn't have to.
>>
>> Thoughts?
>>
>> Thanks!
>> Paul
>>
>>
>
>
> --
> From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT is
> explicitly specified
>

Mime
View raw message