spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From raja kbv <raja...@yahoo.com.INVALID>
Subject How to Parse & flatten JSON object in a text file using Spark & Scala into Dataframe
Date Tue, 22 Dec 2015 18:21:26 GMT
Hi,
I am new to spark.
 I have a text filewith below structure.
 (employeeID: Int, Name: String, ProjectDetails:JsonObject{[{ProjectName, Description, Duriation,
Role}]})Eg:(123456, Employee1, {“ProjectDetails”:[                                                        {“ProjectName”:
“Web Develoement”, “Description” : “Online Sales website”,“Duration” : “6
Months” , “Role” : “Developer”}                                                        
{ “ProjectName”: “Spark Develoement”, “Description” : “Online SalesAnalysis”,
“Duration” : “6 Months” , “Role” : “Data Engineer”}                                                        
{ “ProjectName”: “Scala Training”, “Description” : “Training”,“Duration”
: “1 Month” }                                                         ]                                               }  Could
someone help me to parse & flatten the record asbelow dataframe using scala? employeeID,Name,
ProjectName, Description, Duration, Role123456, Employee1, Web Develoement, Online Sales website,
6Months , Developer123456, Employee1, Spark Develoement, Online Sales Analysis,6 Months, Data
Engineer123456, Employee1, Scala Training, Training, 1 Month, null 
Thank you in advance.
Regards,Raja
Mime
View raw message