spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Date Tue, 22 Nov 2016 05:34:46 GMT
Hi Michael,

I only see spark 2.0.2 which is what I am using currently. Any idea on when
2.1 will be released?

Thanks,
kant

On Mon, Nov 21, 2016 at 5:12 PM, Michael Armbrust <michael@databricks.com>
wrote:

> In Spark 2.1 we've added a from_json
> <https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
> function that I think will do what you want.
>
> On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <kanth909@gmail.com> wrote:
>
>> This seem to work
>>
>> import org.apache.spark.sql._
>> val rdd = df2.rdd.map { case Row(j: String) => j }
>> spark.read.json(rdd).show()
>>
>> However I wonder if this any inefficiency here ? since I have to apply
>> this function for billion rows.
>>
>>
>

Mime
View raw message