spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brandon Geise <brandonge...@gmail.com>
Subject Re: CSV parser - how to parse column containing json data
Date Tue, 02 Oct 2018 21:45:11 GMT
Do your schema inference and then apply the JSON schema using withColumn overwriting the String
representation

 

From: Nirav Patel <npatel@xactlycorp.com>
Date: Tuesday, October 2, 2018 at 5:00 PM
To: <brandongeise@gmail.com>
Cc: spark users <user@spark.apache.org>
Subject: Re: CSV parser - how to parse column containing json data

 

I need to inferSchema from CSV as well. As per your solution, I am creating SructType only
for Json field. So how am I going to mix and match here? i.e. do type inference for all fields
but json field and use custom json_schema for json field. 

 

 

 

 

 

On Thu, Aug 30, 2018 at 5:29 PM Brandon Geise <brandongeise@gmail.com> wrote:

If you know your json schema you can create a struct and then apply that using from_json:

 

val json_schema = StructType(Array(StructField(“x”, StringType, true), StructField(“y”,
StringType, true), StructField(“z”, IntegerType, true)))

 

.withColumn("_c3", from_json(col("_c3_signals"),json_schema))

 

From: Nirav Patel <npatel@xactlycorp.com>
Date: Thursday, August 30, 2018 at 7:19 PM
To: spark users <user@spark.apache.org>
Subject: CSV parser - how to parse column containing json data

 

Is there a way to parse csv file with some column in middle containing json data structure?

 

"a",102,"c","{"x":"xx","y":false,"z":123}","d","e",102.2

 

 

Thanks,

Nirav






        






        


Mime
View raw message