spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-1332) Improve Spark Streaming's Network Receiver and InputDStream API for future stability
Date Tue, 01 Apr 2014 20:52:25 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tathagata Das updated SPARK-1332:
---------------------------------

    Priority: Blocker  (was: Critical)

> Improve Spark Streaming's Network Receiver and InputDStream API for future stability
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-1332
>                 URL: https://issues.apache.org/jira/browse/SPARK-1332
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 0.9.0
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
>            Priority: Blocker
>
> The current Network Receiver API makes it slightly complicated to right a new receiver
as one needs to create an instance of BlockGenerator as shown in SocketReceiver 
> https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala#L51
> Exposing the BlockGenerator interface has made it harder to improve the receiving process.
The API of NetworkReceiver (which was not a very stable API anyways) needs to be change if
we are to ensure future stability. 
> Additionally, the functions like streamingContext.socketStream that create input streams,
return DStream objects. That makes it hard to expose functionality (say, rate limits) unique
to input dstreams. They should return InputDStream or NetworkInputDStream.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message