spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yong Zhang <>
Subject Re: Java to show struct field from a Dataframe
Date Sat, 17 Dec 2016 23:14:48 GMT
"[D" type means a double array type. So this error simple means you have double[] data, but
Spark needs to cast it to Double, as your schema defined.

The error message clearly indicates the data doesn't match with  the type specified in the

I wonder how you are so sure about your data? Do you check it under other tool?


From: Richard Xin <>
Sent: Saturday, December 17, 2016 10:56 AM
To:; user
Subject: Re: Java to show struct field from a Dataframe

data is good

On Saturday, December 17, 2016 11:50 PM, "" <> wrote:

I think the causation is your invanlid Double data , have u checked your data ?


From: Richard Xin<>
Date: 2016-12-17 23:28
To: User<>
Subject: Java to show struct field from a Dataframe
let's say I have a DataFrame with schema of followings:
 |-- name: string (nullable = true)
 |-- location: struct (nullable = true)
 |    |-- longitude: double (nullable = true)
 |    |-- latitude: double (nullable = true); throws following exception:

java.lang.ClassCastException: [D cannot be cast to java.lang.Double
    at scala.runtime.BoxesRunTime.unboxToDouble(
    at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getDouble(rows.scala:44)
    at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getDouble(rows.scala:221)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown

Any advise?
Thanks in advance.

View raw message