spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artur Sukhenko <artur.sukhe...@gmail.com>
Subject 3 equalTo "3.15" = true
Date Wed, 06 Feb 2019 15:31:32 GMT
Hello guys,
I am migrating from Spark 1.6 to 2.2 and have this issue:
I am casting string to short and comparing them with equal .
Original code is:
... when(col(fieldName).equalTo(castedValueCol), castedValueCol).

  otherwise(defaultErrorValueCol)

Reproduce (version 2.3.0.cloudera4):
scala> val df = Seq("3.15").toDF("tier_id")
df: org.apache.spark.sql.DataFrame = [tier_id: string]

scala> val colShort = col("tier_id").cast(ShortType)
colShort: org.apache.spark.sql.Column = CAST(tier_id AS SMALLINT)

scala> val colString = col("tier_id")
colString: org.apache.spark.sql.Column = tier_id

scala> res4.select(colString, colShort, colShort.equalTo(colString)).show
+-------+-------+-------------------------------------+
|tier_id|tier_id|(CAST(tier_id AS SMALLINT) = tier_id)|
+-------+-------+-------------------------------------+
|   3.15|      3|                                 true|
+-------+-------+-------------------------------------+
scala>

Why is this?
-- 
--
Artur Sukhenko

Mime
View raw message