spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabor Somogyi (JIRA)" <>
Subject [jira] [Commented] (SPARK-26213) Custom Receiver for Structured streaming
Date Sat, 01 Dec 2018 16:14:00 GMT


Gabor Somogyi commented on SPARK-26213:

On the other hand looks like this is not a feature suggestion but a question to the community.
In such cases you can write a mail to [|].

If you agree please close the jira.


> Custom Receiver for Structured streaming
> ----------------------------------------
>                 Key: SPARK-26213
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Aarthi
>            Priority: Major
> Hi,
> I have implemented a Custom Receiver for a https/json data source by implementing the
Receievr abstract class as provided in the documentation here []
> This approach works on Spark streaming context  where the custom receiver class is passed
it receiverStream. However I would like the implement the same for Structured streaming as
each of the DStreams have a complex structure and need to be joined with each other based
on complex rules. ([])
>  Structured streaming uses the Spark Session object that takes in DataStreamReader which
is a final class. Please advice on how to implement the custom receiver for Strucutred Streaming. 
> Thanks,

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message