spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arnav kumar <akns2...@gmail.com>
Subject Type Casting Error in Spark Data Frame
Date Mon, 29 Jan 2018 21:26:27 GMT
Hello Experts,

I would need your advice in resolving the below issue when I am trying to
retrieving the data from a dataframe.

Can you please let me know where I am going wrong.

code :


// create the dataframe by parsing the json
// Message Helper describes the JSON Struct
//data out is the json string received from Streaming Engine.

val dataDF = sparkSession.createDataFrame(dataOut, MessageHelper.sqlMapping)
dataDF.printSchema()
/* -- out put of dataDF.printSchema

root
 |-- messageID: string (nullable = true)
 |-- messageType: string (nullable = true)
 |-- meta: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- messageParsedTimestamp: string (nullable = true)
 |    |    |-- ipaddress: string (nullable = true)
 |-- messageData: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- packetID: string (nullable = true)
 |    |    |-- messageID: string (nullable = true)
 |    |    |-- unixTime: string (nullable = true)



*/


dataDF.createOrReplaceTempView("message")
val routeEventDF=sparkSession.sql("select messageId
,messageData.unixTime,messageData.packetID, messageData.messageID from
message")
routeEventDF.show


Error  on routeEventDF.show
Caused by: java.lang.RuntimeException:
org.apache.spark.sql.catalyst.expressions.GenericRow is not a valid
external type for schema of
array<struct<messageParsedTimestamp:string,ipaddress:string,port:string,message:string>>>>
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.evalIfFalseExpr14$(Unknown
Source)


Appreciate your help

Best Regards
Arnav Kumar.

Mime
View raw message