spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Kronenfeld <nkronenf...@uncharted.software>
Subject Migration issue upgrading from Spark 2.4.5 to spark 3.0.1
Date Thu, 10 Dec 2020 20:10:45 GMT
Hi, everyone.

We are in the process of upgrading our Spark version, and are hitting what
appears to be the bug described in
https://issues.apache.org/jira/browse/SPARK-29497 - namely, that when we
pass anonymous functions to the workers, we get an error

Caused by: java.lang.ClassCastException: cannot assign instance
    of java.lang.invoke.SerializedLambda to field <XYZ>
    of type scala.Function1 in instance of <xyz>

As we are using Gradle instead of Maven, the suggested fixes there, even as
far as they go, don't work for us.

Has anyone else seen this? Does anyone have any suggestions on how to get
around it, other than to stop passing around anonymous functions?

Thanks,

            - Nathan Kronenfeld
            - Uncharted Software

Mime
View raw message