spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Zhemzhitsky <szh.s...@gmail.com>
Subject Accumulators of Spark 1.x no longer work with Spark 2.x
Date Thu, 15 Mar 2018 21:55:10 GMT
Hi there,

I've noticed that accumulators of Spark 1.x no longer work with Spark
2.x failing with
java.lang.AssertionError: assertion failed: copyAndReset must return a
zero value copy

It happens while serializing an accumulator here [1] although
copyAndReset returns zero-value copy for sure, just consider the
accumulator below

val concatParam = new AccumulatorParam[jl.StringBuilder] {
  override def zero(initialValue: jl.StringBuilder): jl.StringBuilder
= new jl.StringBuilder()
  override def addInPlace(r1: jl.StringBuilder, r2: jl.StringBuilder):
jl.StringBuilder = r1.append(r2)
}

So, Spark treats zero value as non-zero due to how isZero [2] is
implemented in LegacyAccumulatorWrapper

override def isZero: Boolean = _value == param.zero(initialValue)

All this means, that the values to be accumulated must implement
equals and hashCode, otherwise isZero is very likely to always return
false.

So I'm wondering why this assertion is necessary and whether it can be
safely removed from there?

[1] https://github.com/apache/spark/blob/4f5bad615b47d743b8932aea1071652293981604/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L165
[2] https://github.com/apache/spark/blob/4f5bad615b47d743b8932aea1071652293981604/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L489
[3] https://issues.apache.org/jira/browse/SPARK-23697

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message