spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Straub (Jira)" <>
Subject [jira] [Created] (SPARK-30270) Can't pickle abstract classes (with cloudpickle)
Date Mon, 16 Dec 2019 13:51:00 GMT
Sebastian Straub created SPARK-30270:

             Summary: Can't pickle abstract classes (with cloudpickle)
                 Key: SPARK-30270
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.4.4
            Reporter: Sebastian Straub

I can't use any classes that are derived from abstract classes in PySpark, because cloudpickle 
can't pickle them. Example:

import pickle
from abc import ABC
from pyspark import cloudpickle

class Foo(ABC):

class Bar(Foo):

bar = Bar()

# pickle dump works fine
# cloudpickle doesn't
A similar bug has already been reported in SPARK-21439 and marked resolved, but I can confirm
that the issue still persists.


This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message