spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <>
Subject Re: [Query] Columnar transformation without Structured Streaming
Date Sun, 01 Apr 2018 18:20:17 GMT

as far as I understand, given my limited experience with streaming I may be
wrong, DStreams are row based data and in case we want to transform them
 to columnar based data storage then there is a computation overhead. That
may be one of the reasons why its better to avoid.

On other hand, I am a bit curious, why would you want to understand about
dstreams in case you have picked up from structured streaming?

Gourav Sengupta

On Thu, Mar 29, 2018 at 1:41 PM, Aakash Basu <>

> Hi,
> I started my Spark Streaming journey from Structured Streaming using Spark
> 2.3, where I can easily do Spark SQL transformations on streaming data.
> But, I want to know, how can I do columnar transformation (like, running
> aggregation or casting, et al) using the prior utility of DStreams? Is
> there a way? Do I have to use map on RDD and go about the complex
> transformative steps? Or can I convert a DStream into DF and do the job?
> Appreciations in advance!
> Thanks,
> Aakash.

View raw message