spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <gourav.sengu...@gmail.com>
Subject Re: [Query] Columnar transformation without Structured Streaming
Date Sun, 01 Apr 2018 18:20:17 GMT
Hi,

as far as I understand, given my limited experience with streaming I may be
wrong, DStreams are row based data and in case we want to transform them
 to columnar based data storage then there is a computation overhead. That
may be one of the reasons why its better to avoid.

On other hand, I am a bit curious, why would you want to understand about
dstreams in case you have picked up from structured streaming?


Regards,
Gourav Sengupta

On Thu, Mar 29, 2018 at 1:41 PM, Aakash Basu <aakash.spark.raj@gmail.com>
wrote:

> Hi,
>
> I started my Spark Streaming journey from Structured Streaming using Spark
> 2.3, where I can easily do Spark SQL transformations on streaming data.
>
> But, I want to know, how can I do columnar transformation (like, running
> aggregation or casting, et al) using the prior utility of DStreams? Is
> there a way? Do I have to use map on RDD and go about the complex
> transformative steps? Or can I convert a DStream into DF and do the job?
>
> Appreciations in advance!
>
> Thanks,
> Aakash.
>

Mime
View raw message