spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly
Date Fri, 01 Feb 2019 19:31:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shixiong Zhu resolved SPARK-26806.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 3.0.0
                   2.4.1
                   2.3.3
                   2.2.4

> EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly
> ----------------------------------------------------------------
>
>                 Key: SPARK-26806
>                 URL: https://issues.apache.org/jira/browse/SPARK-26806
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.2.1, 2.3.0, 2.3.1, 2.3.2, 2.4.0
>            Reporter: liancheng
>            Assignee: Shixiong Zhu
>            Priority: Major
>             Fix For: 2.2.4, 2.3.3, 2.4.1, 3.0.0
>
>
> Right now, EventTimeStats.merge doesn't handle "zero.merge(zero)". This will make "avg"
become "NaN". And whatever gets merged with the result of "zero.merge(zero)", "avg" will still
be "NaN". Then finally, "NaN".toLong will return "0" and the user will see the following incorrect
report:
> {code}
> "eventTime" : {
>     "avg" : "1970-01-01T00:00:00.000Z",
>     "max" : "2019-01-31T12:57:00.000Z",
>     "min" : "2019-01-30T18:44:04.000Z",
>     "watermark" : "1970-01-01T00:00:00.000Z"
>   }
> {code}
> This issue was reported by [~liancheng]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message