spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yong Zhang <java8...@hotmail.com>
Subject Re: Java to show struct field from a Dataframe
Date Sat, 17 Dec 2016 23:14:48 GMT
"[D" type means a double array type. So this error simple means you have double[] data, but
Spark needs to cast it to Double, as your schema defined.


The error message clearly indicates the data doesn't match with  the type specified in the
schema.


I wonder how you are so sure about your data? Do you check it under other tool?


Yong


________________________________
From: Richard Xin <richardxin168@yahoo.com.INVALID>
Sent: Saturday, December 17, 2016 10:56 AM
To: zjp_jdev@163.com; user
Subject: Re: Java to show struct field from a Dataframe

data is good


On Saturday, December 17, 2016 11:50 PM, "zjp_jdev@163.com" <zjp_jdev@163.com> wrote:


I think the causation is your invanlid Double data , have u checked your data ?

________________________________
zjp_jdev@163.com

From: Richard Xin<mailto:richardxin168@yahoo.com.INVALID>
Date: 2016-12-17 23:28
To: User<mailto:user@spark.apache.org>
Subject: Java to show struct field from a Dataframe
let's say I have a DataFrame with schema of followings:
root
 |-- name: string (nullable = true)
 |-- location: struct (nullable = true)
 |    |-- longitude: double (nullable = true)
 |    |-- latitude: double (nullable = true)

df.show(); throws following exception:

java.lang.ClassCastException: [D cannot be cast to java.lang.Double
    at scala.runtime.BoxesRunTime.unboxToDouble(BoxesRunTime.java:119)
    at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getDouble(rows.scala:44)
    at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getDouble(rows.scala:221)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
Source)
....

Any advise?
Thanks in advance.
Richard



Mime
View raw message