spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jay vyas <jayunit100.apa...@gmail.com>
Subject Streams: How do RDDs get Aggregated?
Date Sat, 11 Oct 2014 19:30:11 GMT
Hi spark !

I dont quite yet understand the semantics of RDDs in a streaming context
very well yet.

Are there any examples of how to implement CustomInputDStreams, with
corresponding Receivers in the docs ?

Ive hacked together a  custom stream, which is being opened and is
consuming data internally, however,  it is not empty RDDs, even though I am
calling store(...) mutliple times - however, Im relying on the default
implementation of store(...) which may be a mistake on my end.

By making my slide duration  small, I can make sure that indeed the job
finishes - however im not quite sure how we're supposed to shuttle data
from the ReceiverInputDStream into RDDs ?

Thanks!

Mime
View raw message