spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Takeshi Yamamuro (Jira)" <>
Subject [jira] [Created] (SPARK-30998) ClassCastException when a generator having nested inner generators
Date Sun, 01 Mar 2020 05:03:00 GMT
Takeshi Yamamuro created SPARK-30998:

             Summary: ClassCastException when a generator having nested inner generators
                 Key: SPARK-30998
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.0.0, 3.1.0, 2.4.6
            Reporter: Takeshi Yamamuro

A query below failed in the master/branch-3.0/branch-2.4;

scala> sql("select array(array(1, 2), array(3)) ar").select(explode(explode($"ar"))).show()
20/03/01 13:51:56 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1]
java.lang.ClassCastException: scala.collection.mutable.ArrayOps$ofRef cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData
	at org.apache.spark.sql.catalyst.expressions.ExplodeBase.eval(generators.scala:313)
	at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$8(GenerateExec.scala:108)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
	at scala.collection.Iterator$ConcatIterator.hasNext(Iterator.scala:222)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message