spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eddie" <eddie.pantri...@gmail.com>
Subject [Advanced][Spark Core][Debug] Spark incorrectly deserializes object during Dataset.map
Date Sat, 25 Sep 2021 00:02:32 GMT
I am calling the Spark Dataset API (map method) and getting exceptions on deserialization of
task results. I am calling this API from Clojure using standard JVM interop syntax.

This gist has a tiny Clojure program that shows the problem, as well as the corresponding
(working) Scala implementation. There is also a full stack trace and Spark logs for a run
of the Clojure code. https://gist.github.com/erp12/233a60574dc157aa544079959108f9db

Interestingly, the exception is not raised if the Clojure code is first compiled to bytecode,
or if it is executed from a REPL. It seems that, depending on the context, the Spark deserializer
is improperly deserializing a Function3 as a SerializedLambda. 

Does anyone have any ideas on why is this happening? Is this a Spark bug?
Mime
View raw message