spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eran Witkon <eranwit...@gmail.com>
Subject Re: How to Parse & flatten JSON object in a text file using Spark & Scala into Dataframe
Date Wed, 23 Dec 2015 14:46:48 GMT
Did you get a solution for this?
On Tue, 22 Dec 2015 at 20:24 raja kbv <rajakbv@yahoo.com.invalid> wrote:

> Hi,
>
> I am new to spark.
>
> I have a text file with below structure.
>
>
> (employeeID: Int, Name: String, ProjectDetails: JsonObject{[{ProjectName,
> Description, Duriation, Role}]})
> Eg:
> (123456, Employee1, {“ProjectDetails”:[
>                                                          { “ProjectName”:
> “Web Develoement”, “Description” : “Online Sales website”, “Duration”
: “6
> Months” , “Role” : “Developer”}
>                                                          { “ProjectName”:
> “Spark Develoement”, “Description” : “Online Sales Analysis”, “Duration”
:
> “6 Months” , “Role” : “Data Engineer”}
>                                                          { “ProjectName”:
> “Scala Training”, “Description” : “Training”, “Duration” : “1 Month”
}
>                                                           ]
>                                                 }
>
>
> Could someone help me to parse & flatten the record as below dataframe
> using scala?
>
> employeeID,Name, ProjectName, Description, Duration, Role
> 123456, Employee1, Web Develoement, Online Sales website, 6 Months ,
> Developer
> 123456, Employee1, Spark Develoement, Online Sales Analysis, 6 Months,
> Data Engineer
> 123456, Employee1, Scala Training, Training, 1 Month, null
>
>
> Thank you in advance.
>
> Regards,
> Raja
>

Mime
View raw message