spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 周康 <zhoukang199...@gmail.com>
Subject Re: Spark SQL unexpected behavior when comparing timestamp to date
Date Thu, 08 Mar 2018 14:05:57 GMT
I have added some comment on the related jira.

2018-03-03 2:28 GMT+08:00 Dong Jiang <djiang@dataxu.com>:

> Hi,
>
> I opened a JIRA ticket https://issues.apache.org/jira/browse/SPARK-23549,
> I
> don't know if anyone can take a look?
>
> Spark SQL unexpected behavior when comparing timestamp to date
>
> scala> spark.version
>
> res1: String = 2.2.1
>
> scala> spark.sql("select cast('2017-03-01 00:00:00' as timestamp) between
> cast('2017-02-28' as date) and cast('2017-03-01' as date)").show
>
>
> +-----------------------------------------------------------
> ------------------------------------------------------------
> ------------------------------------------------------------
> ----------------------------+
>
> |((CAST(CAST(2017-03-01 00:00:00 AS TIMESTAMP) AS STRING) >=
> CAST(CAST(2017-02-28 AS DATE) AS STRING)) AND (CAST(CAST(2017-03-01
> 00:00:00
> AS TIMESTAMP) AS STRING) <= CAST(CAST(2017-03-01 AS DATE) AS STRING)))|
>
> +-----------------------------------------------------------
> ------------------------------------------------------------
> ------------------------------------------------------------
> ----------------------------+
>
> |
> false|
>
> +-----------------------------------------------------------
> ------------------------------------------------------------
> ------------------------------------------------------------
> ----------------------------+
>
> As shown above, when a timestamp is compared to date in SparkSQL, both
> timestamp and date are downcast to string, and leading to unexpected
> result.
> If run the same SQL in presto/Athena, I got the expected result
>
> select cast('2017-03-01 00:00:00' as timestamp) between cast('2017-02-28'
> as
> date) and cast('2017-03-01' as date)
>         _col0
> 1       true
>
> Is this a bug for Spark or a feature?
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message