spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: How to convert Dstream of JsonObject to Dataframe in spark 2.1.0?
Date Mon, 24 Apr 2017 20:02:55 GMT
Thanks sam!


On Mon, Apr 24, 2017 at 1:50 AM, Sam Elamin <hussam.elamin@gmail.com> wrote:

> you have 2 options
> 1 )Clean ->Write your own parser to through each property and create a
> dataset
> 2) Hacky but simple -> Convert to json string then read in using
> spark.read.json(jsonString)
>
> Please bear in mind the second option is expensive which is why it is hacky
>
> I wrote my own parser here
> <https://github.com/samelamin/spark-bigquery/blob/master/src/main/scala/com/samelamin/spark/bigquery/converters/SchemaConverters.scala>
> which you can use to convert between JsonObjects to StructType schemas
>
> Regards
> Sam
>
>
> On Sun, Apr 23, 2017 at 7:50 PM, kant kodali <kanth909@gmail.com> wrote:
>
>> Hi All,
>>
>> How to convert Dstream of JsonObject to Dataframe in spark 2.1.0? That
>> JsonObject is from Gson Library.
>>
>> Thanks!
>>
>
>

Mime
View raw message