spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Straub (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-30270) Can't pickle abstract classes (with cloudpickle)
Date Mon, 16 Dec 2019 13:51:00 GMT
Sebastian Straub created SPARK-30270:
----------------------------------------

             Summary: Can't pickle abstract classes (with cloudpickle)
                 Key: SPARK-30270
                 URL: https://issues.apache.org/jira/browse/SPARK-30270
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.4.4
            Reporter: Sebastian Straub


I can't use any classes that are derived from abstract classes in PySpark, because cloudpickle 
can't pickle them. Example:

 
{code:java}
import pickle
from abc import ABC
from pyspark import cloudpickle


class Foo(ABC):
    pass

class Bar(Foo):
    pass

bar = Bar()

# pickle dump works fine
pickle.dumps(bar)
# cloudpickle doesn't
cloudpickle.dumps(bar)
{code}
A similar bug has already been reported in SPARK-21439 and marked resolved, but I can confirm
that the issue still persists.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message