spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rsearle <eggsea...@verizon.net>
Subject Spark updateStateByKey fails with class leak when using case classes - resend
Date Thu, 07 May 2015 02:48:56 GMT

<<Apologies for the repeat. The first was rejected by the submission
process>>

I created a simple Spark streaming program using updateStateByKey.
The domain is represented by case classes for clarity, type safety, etc.

Spark job continuously loads new classes, which are removed by GC to
maintain
a relatively constant level of active classes instances. The total memory 
footprint grows and the throughput slows, until the job fails. The failure
is generally triggered when
the processing cannot no longer keep up with the rate limited input. 

The offending classes are of the form
"sun.reflect.GeneratedSerializationConstructorAccessor125".

This failure does not occur if the job is rewritten to use only Scala
classes: tuples and primitives.

The following github project contains the test code and more details

https://github.com/searler/SparkStreamingLeak
<https://github.com/searler/SparkStreamingLeak>  




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-updateStateByKey-fails-with-class-leak-when-using-case-classes-resend-tp22793.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message