spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-26962) Windows Function LEAD in Spark SQL is not fetching consistent results.
Date Fri, 22 Feb 2019 07:31:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-26962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774865#comment-16774865
] 

Hyukjin Kwon commented on SPARK-26962:
--------------------------------------

Are you able to show the results in the JIRA description? It would be awesome to narrow down
and find the root condition that it reads the same data as well.

> Windows Function LEAD in Spark SQL is not fetching consistent results.
> ----------------------------------------------------------------------
>
>                 Key: SPARK-26962
>                 URL: https://issues.apache.org/jira/browse/SPARK-26962
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: Shiva Sankari Perambalam
>            Priority: Major
>
> Using a Lead function on a DATETIME column is giving inconsistent results in Spark
sql.
> {code:java}
> Lead(date) over (partition by id, code order by date){code}
> where Date is DATETIME, id and code a String.
> {code:java}
> val testdf1= sparkSession.sql(s""" select date, lead(date) over (partition by id, code
order by date) as lead_date from foo"""){code}
> The result set is sometimes having the same data as the date instead of the lead_date
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message