spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: How do I flatten JSON blobs into a Data Frame using Spark/Spark SQL
Date Tue, 22 Nov 2016 01:12:43 GMT
In Spark 2.1 we've added a from_json
<https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2902>
function that I think will do what you want.

On Fri, Nov 18, 2016 at 2:29 AM, kant kodali <kanth909@gmail.com> wrote:

> This seem to work
>
> import org.apache.spark.sql._
> val rdd = df2.rdd.map { case Row(j: String) => j }
> spark.read.json(rdd).show()
>
> However I wonder if this any inefficiency here ? since I have to apply
> this function for billion rows.
>
>

Mime
View raw message