spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eddie" <>
Subject [Advanced][Spark Core][Debug] Spark incorrectly deserializes object during
Date Sat, 25 Sep 2021 00:02:32 GMT
I am calling the Spark Dataset API (map method) and getting exceptions on deserialization of
task results. I am calling this API from Clojure using standard JVM interop syntax.

This gist has a tiny Clojure program that shows the problem, as well as the corresponding
(working) Scala implementation. There is also a full stack trace and Spark logs for a run
of the Clojure code.

Interestingly, the exception is not raised if the Clojure code is first compiled to bytecode,
or if it is executed from a REPL. It seems that, depending on the context, the Spark deserializer
is improperly deserializing a Function3 as a SerializedLambda. 

Does anyone have any ideas on why is this happening? Is this a Spark bug?
View raw message