spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Max Kießling (JIRA) <>
Subject [jira] [Created] (SPARK-23610) Cast of ArrayType of NullType to ArrayType of nullable material type does not work
Date Tue, 06 Mar 2018 10:10:00 GMT
Max Kießling created SPARK-23610:

             Summary: Cast of ArrayType of NullType to ArrayType of nullable material type
does not work
                 Key: SPARK-23610
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.2.0
            Reporter: Max Kießling

Given a DataFrame that contains a column with _ArrayType of NullType_
casting this column into ArrayType of any material nullable type (e.g. _ArrayType(LongType,
true)_ ) should be possible.
it("can cast arrays of null type into arrays of nullable material types") {
 val inputData = Seq(

val schema = StructType(Seq(
 StructField("list", ArrayType(NullType, true), false)

val data = caps.sparkSession.createDataFrame(inputData, schema)

data.withColumn("longList",data.col("list").cast(ArrayType(LongType, true))).show

This test fails with the message: 

NullType (of class org.apache.spark.sql.types.NullType$)
 scala.MatchError: NullType (of class org.apache.spark.sql.types.NullType$)
 at org.apache.spark.sql.catalyst.expressions.Cast.castToLong(Cast.scala:310)
 at org.apache.spark.sql.catalyst.expressions.Cast.castArray(Cast.scala:455)
 at org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:531)
 at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:531)
 at org.apache.spark.sql.catalyst.expressions.Cast.nullSafeEval(Cast.scala:533)
 at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:327)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message