spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Terry Kim (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-29664) Column.getItem behavior is not consistent with Scala version
Date Wed, 30 Oct 2019 16:58:00 GMT
Terry Kim created SPARK-29664:
---------------------------------

             Summary: Column.getItem behavior is not consistent with Scala version
                 Key: SPARK-29664
                 URL: https://issues.apache.org/jira/browse/SPARK-29664
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 3.0.0
            Reporter: Terry Kim


In PySpark, Column.getItem's behavior is different from the Scala version.

For example,
In PySpark:
{code:python}
df = spark.range(2)
map_col = create_map(lit(0), lit(100), lit(1), lit(200))
df.withColumn("mapped", map_col.getItem(col('id'))).show()
# +---+------+
# | id|mapped|
# +---+------+
# |  0|   100|
# |  1|   200|
# +---+------+
{code}

In Scala:
{code:scala}
val df = spark.range(2)
val map_col = map(lit(0), lit(100), lit(1), lit(200))
// The following getItem results in the following exception, which is the right behavior:
// java.lang.RuntimeException: Unsupported literal type class org.apache.spark.sql.Column
id
//  at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:78)
//  at org.apache.spark.sql.Column.getItem(Column.scala:856)
//  ... 49 elided
df.withColumn("mapped", map_col.getItem(col("id"))).show


// You have to use apply() to match with PySpark's behavior.
df.withColumn("mapped", map_col(col("id"))).show
// +---+------+
// | id|mapped|
// +---+------+
// |  0|   100|
// |  1|   200|
// +---+------+
{code}

Looking at the code for Scala implementation, PySpark's behavior is incorrect since the argument
to getItem becomes `Literal`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message