spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prashant Sharma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-15227) InputStream stop-start semantics + empty implementations
Date Tue, 17 May 2016 05:44:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15286089#comment-15286089
] 

Prashant Sharma commented on SPARK-15227:
-----------------------------------------

If start and stop are overridden by a particular DStream, they are called when the streams
are started(to do some intialization) and stopped (to do so cleanup). However, if there is
nothing to initialize and cleanup - then they can be left empty. 

Pause and resume is very different from start and stop. For example, if you pause - what happens
to the incoming stream. They are buffered or they are dropped ? Those semantics need to be
discussed, before we can talk about that. It is possible to implement it by having a custom
receiver.

I am not sure, but since the development efforts are shifted towards the structured streaming,
it will be interesting to see - how this sort of thing gets implemented.



> InputStream stop-start semantics + empty implementations
> --------------------------------------------------------
>
>                 Key: SPARK-15227
>                 URL: https://issues.apache.org/jira/browse/SPARK-15227
>             Project: Spark
>          Issue Type: Improvement
>          Components: Input/Output, Streaming
>    Affects Versions: 1.6.1
>            Reporter: Stas Levin
>            Priority: Minor
>
> Hi,
> Seems like quite a few InputStream(s) currently leave the start and stop methods empty.

> I was hoping to hear your thoughts on:
> 1. Whether there were any particular reasons to leave these methods empty ?
> 2. Do the stop/start semantics of InputStream(s) aim to support pause-resume use-cases,
or is it a one way ticket? 
> A pause-resume kind of thing could be really useful for cases where one wishes to load
new offline data for the streaming app to run on top of, without restarting the entire app.
> Thanks a lot,
> Stas



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message