spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabor Somogyi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-26213) Custom Receiver for Structured streaming
Date Sat, 01 Dec 2018 16:10:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-26213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705884#comment-16705884
] 

Gabor Somogyi commented on SPARK-26213:
---------------------------------------

Structured streaming doesn't have receiver concept. If you take a look at how DStreams evolved
a new direct approach has been introduced.
 On the other hand you can create a custom source by implementing [this trait|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Source.scala].
There are several examples.

 

> Custom Receiver for Structured streaming
> ----------------------------------------
>
>                 Key: SPARK-26213
>                 URL: https://issues.apache.org/jira/browse/SPARK-26213
>             Project: Spark
>          Issue Type: New Feature
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Aarthi
>            Priority: Major
>
> Hi,
> I have implemented a Custom Receiver for a https/json data source by implementing the
Receievr abstract class as provided in the documentation here [https://spark.apache.org/docs/latest//streaming-custom-receivers.html]
> This approach works on Spark streaming context  where the custom receiver class is passed
it receiverStream. However I would like the implement the same for Structured streaming as
each of the DStreams have a complex structure and need to be joined with each other based
on complex rules. ([https://stackoverflow.com/questions/53449599/join-two-spark-dstreams-with-complex-nested-structure])
>  Structured streaming uses the Spark Session object that takes in DataStreamReader which
is a final class. Please advice on how to implement the custom receiver for Strucutred Streaming. 
> Thanks,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message