spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different
Date Sat, 27 Jul 2019 18:29:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894524#comment-16894524
] 

Sean Owen commented on SPARK-28519:
-----------------------------------

A summary of the dev@ discussion:

This is almost surely because the java.lang.Math by default defers to java.lang.StrictMath
implementations, but in the Oracle JDKs, calls some x86 intrinsic for things like log() and
pow(). That is, I think we'd find this happens in OpenJDK too, but haven't confirmed.

atanh isn't implemented in java.lang.Math, but uses log(): {{0.5 * math.log((1 + x) / (1 -
x))}}. While we can 'fix' the variation by directly using StrictMath.log(), we can also just
use a different formula that should also happen to be slightly more accurate for small x:
{{0.5 * ((math.log1p(x) - math.log1p(-x))}}. I suggest this.

The other result involving pow() is just a more direct illustration of the platform variation.
There we can't just adjust a formula. We can choose to implement pow() with StrictMath.pow()
directly. This makes the result the same across platforms. It might cost a little performance,
but this probably is trivial in Spark. The only downside is one would find that Spark's pow()
may then return a 'right' answer but not the same one you'd find, exactly, by just calling
pow() in Java!

Really then the same argument goes for log(). It's odd that log(3) for example gives a different
result on different platforms (on Spark and in the JVM). It feels like we should require StrictMath
for this.

> Tests failed on aarch64 due the value of math.log and power function is different
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-28519
>                 URL: https://issues.apache.org/jira/browse/SPARK-28519
>             Project: Spark
>          Issue Type: Test
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: huangtianhua
>            Priority: Major
>
> Sorry to disturb again, we ran unit tests on arm64 instance, and there are other sql
tests failed:
> {code}
>  - pgSQL/float8.sql *** FAILED ***
>  Expected "{color:#f691b2}0.549306144334054[9]{color}", but got "{color:#f691b2}0.549306144334054[8]{color}"
Result did not match for query #56
>  SELECT atanh(double('0.5')) (SQLQueryTestSuite.scala:362)
>  - pgSQL/numeric.sql *** FAILED ***
>  Expected "2 {color:#59afe1}2247902679199174[72{color} 224790267919917955.1326161858
>  4 7405685069595001 7405685069594999.0773399947
>  5 5068226527.321263 5068226527.3212726541
>  6 281839893606.99365 281839893606.9937234336
>  7 {color:#d04437}1716699575118595840{color} 1716699575118597095.4233081991
>  8 167361463828.0749 167361463828.0749132007
>  9 {color:#14892c}107511333880051856]{color} 107511333880052007....", but got "2 {color:#59afe1}2247902679199174[40{color}
224790267919917955.1326161858
>  4 7405685069595001 7405685069594999.0773399947
>  5 5068226527.321263 5068226527.3212726541
>  6 281839893606.99365 281839893606.9937234336
>  7 {color:#d04437}1716699575118595580{color} 1716699575118597095.4233081991
>  8 167361463828.0749 167361463828.0749132007
>  9 {color:#14892c}107511333880051872]{color} 107511333880052007...." Result did not match
for query #496
>  SELECT t1.id1, t1.result, t2.expected
>  FROM num_result t1, num_exp_power_10_ln t2
>  WHERE t1.id1 = t2.id
>  AND t1.result != t2.expected (SQLQueryTestSuite.scala:362)
> {code}
> The first test failed, because the value of math.log(3.0) is different on aarch64:
> # on x86_64:
> {code}
> scala> math.log(3.0)
> res50: Double = 1.0986122886681098
> {code}
> # on aarch64:
> {code}
> scala> math.log(3.0)
> res19: Double = 1.0986122886681096
> {code}
> And I tried {{math.log(4.0)}}, {{math.log(5.0)}} and they are same, I don't know why
{{math.log(3.0)}} is so special? But the result is different indeed on aarch64.
> The second test failed, because some values of pow() is different on aarch64, according
to the test, I took tests on aarch64 and x86_64, take '-83028485' as example:
> # on x86_64:
> {code}
> scala> import java.lang.Math._
> import java.lang.Math._
> scala> abs(-83028485)
> res3: Int = 83028485
> scala> var a = -83028485
> a: Int = -83028485
> scala> abs(a)
> res4: Int = 83028485
> scala> math.log(abs(a))
> res5: Double = 18.234694299654787
> scala> pow(10, math.log(abs(a)))
> res6: Double ={color:#d04437} 1.71669957511859584E18{color}
> {code}
> # on aarch64:
> {code}
> scala> var a = -83028485
> a: Int = -83028485
> scala> abs(a)
> res38: Int = 83028485
> scala> math.log(abs(a))
> res39: Double = 18.234694299654787
> scala> pow(10, math.log(abs(a)))
> res40: Double = 1.71669957511859558E18
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message