spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rsearle <eggsea...@verizon.net>
Subject Re: Spark updateStateByKey fails with class leak when using case classes - resend
Date Wed, 17 Jun 2015 11:21:55 GMT
1.4.0 resolves the problem.

The total classes loaded for an updateStateByKey over Int and String types
does not increase.

The total classes loaded for an updateStateByKey over case classes does
increase over time, but
the processing remains stable. Both memory consumption and CPU load remain
bounded.

The processing time graph for the case class test is much noisier, with
repeated spikes to 2.5x over the base line. 
This is quite clear in the histogram, with two distinct clusters.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-updateStateByKey-fails-with-class-leak-when-using-case-classes-resend-tp22793p23364.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message